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Abstract. Motivated by known examples of global integrals which represent automorphic 
^-functions, this paper initiates the study of a certain two-dimensional array of global 
integrals attached to any reductive algebraic group, indexed by maximal parabolic subgroups 
in one direction and by unipotent conjugacy classes in the other. Fourier coefficients attached 
to unipotent classes, Gelfand-Kirillov dimension of automorphic representations, and an 
identity which, empirically, appears to constrain the unfolding process are presented in 
detail with examples selected from the exceptional groups. Two new Eulerian integrals are 
included among these examples. 
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1. Introduction 

One of the important tools in the study of analytic properties of automorphic L functions 
is the method of integral representations. In this method one seeks to write down a global 
integral which depends on a complex variable s, show that this integral converges for Re(s) 
large, and has a continuation to the whole complex plane. Then the goal is to show that 
the integral is Eulerian, i.e. that it has an Euler product. In the literature, when the global 
integral contains an integration over a reductive group, this method is known as the Rankin- 
Selberg method. In the case when the integral is the Whittaker coefficient of an Eisenstein 
series, this method is known as the Langlands-Shahidi method. From the perspective which 
is adopted in this work, this division is not particularly natural, and hence we prefer to refer 
simply to "the method of integral representations." 

It is an easy task to write down an integral which has an analytic continuation. It is also 
easy to write global integrals which are Eulerian. However, to find integrals which satisfy 
both conditions is quite hard. We are still unaware of a general method of obtaining all 
integrals which satisfy both conditions. 

In this paper we use ideas which were introduced in |G1] and |G3] in order to start a 
certain systematic approach for constructing examples of global integrals. We do it in the 
exceptional groups. There are two main reasons for that. First, the exceptional groups are 
not a part of an infinite family of groups. This enables us to check a finite list of cases. 
Second, the exceptional groups are less studied from these aspects than the classical groups. 
This gives the hope of possibly finding some interesting constructions. However, the methods 
introduced here are also applicable for the classical groups, and we do hope to study this 
issue in the future. 

There are two main ingredients in our approach. The first is to use the classification 
of unipotent orbits (i.e., conjugacy classes consisting of unipotent elements) in order to 
write down global integrals. The main idea is as follows. Given a unipotent orbit O in an 



exceptional group H, we attach to it a parabolic subgroup Pq = MqUq and a character 
ipu a defined on Uo(F)\Uo(&) (or, in some cases, a subgroup). The stabilizer of ipu m 
makes sense as an algebraic group defined over F, and it follows from the results in [C] . 
that the identity component of this stabilizer is a reductive group C. Let E T (h, s) denote an 
Eisenstein series defined on H(A), associated with the induced representation Indp/^s t8 s p . 
See section 4 for details. Let n denote an irreducible cuspidal representation of C(A). Then, 
under certain assumptions on the unipotent orbit O, we can form the global integral 

(1) J J cp^(g)E T (ug,s)ip Uo (u)dudg. 

C(F)\C(A) U (F)\UoW 

See sections 3 and 4 for details. Because of the cuspidality of n, this integral converges for 
Re(s) large and has at least a meromorphic continuation to the whole complex plane. Thus 
it clearly satisfies the first requirement we imposed above on our integrals. Notice that for 
certain unipotent orbits, the group C is trivial. In this case the above integral is just the 
Fourier coefficient of the Eisenstein series. When Uq is the maximal unipotent subgroup of 
H, then ipu is a generic character, and our construction specializes to the Langlands-Shahidi 
construction. 

The next step is to address the problem of when an integral of the form given in equation 
dTJ is Eulerian. This clearly depends on the representations 7r and r. Assume that 7r is 
a generic representation. A sufficient condition for the integral (pQ) to be Eulerian is that 
it is equal to an integral which involves the Whittaker function attached to (p % , and some 
function in a model of r with similar factorization properties, integrated over a factorizable 
domain. We refer to such an integral as a 'Whittaker integral." Experience shows that if 
an integral of the type ([1]) unfolds to Whittaker integral, and is Eulerian, then a certain 
dimension equality holds. See |G3] for some more details. Therefore, we restrict to those 
integrals which satisfy this equation. This is all discussed in detail in section 4. We sketch 
the main ideas. To each of the representations 7r and E T (-,s) we attach a number which we 
refer to as the Gelfand-Kirillov dimension of the representation. The identity we require is 

dim C + dim Uq = dim tc + dim E T (-, s) 

This gives us an equation for dim r, which then gives us certain information of which rep- 
resentation t to choose. To be clear, as far as we know, every global Eulerian integral of 
the type (JTJ), which unfolds to a Whittaker coefficient of 7r satisfies the above dimension 
identity. However, there are global integrals, which satisfy this identity but do not unfold 
to a Whittaker coefficient of tt. There are many such examples; we obtain some in section 

[7J Further, the question of whether an integral of the type ([T]) is Eulerian, and the question 
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of whether it unfolds to a Whittaker coefficient of tt are two related, but separate questions. 
An answer to one does not necessarily determine an answer to the other either way. 

The content of the paper is as follows. After fixing some notations we show how to attach 
a set of Fourier coefficients to a unipotent orbit of an exceptional group. This is done in 
section 3. In section 4 we construct the global integrals we intend to study and give a 
precise description of the basic dimension identity we use. There are several cases, and we 
discuss each of them. At the end of section 4 we write down a table of the condition which 
must be placed on the representation r in each case, in order for the dimension formula 
to be satisfied. We do this for the exceptional group F±. This condition gives information 
about the Fourier coefficients supported by r. It is worthwhile to mention that in four of the 
rows in the table, the group C is trivial. In these cases, the integral ([I]) is just a Fourier 
coefficient of an Eisenstein series. The integrals obtained by considering the last row, are 
the Langlands-Shahidi integrals for the exceptional group F 4 . 

Section 5 is devoted to the general process of unfolding a global integral. We carry out 
this process in detail and as generally as we can. In this way we also fix some notation which 
will be of help to us later on. According to the general outcome, we partition the set of 
integrals into various types. Maybe the most interesting integrals are the ones we refer to 
as open orbit type integrals. See definition |29j These are those integrals, such that after 
some unfolding, we obtain as inner integration, an integral of a similar type to the one we 
started with, and which satisfies a similar dimension identity. This shows that in some sense 
the process is inductive. The study of the global integral is reduced to the study of a global 
integral defined on smaller groups. 

Starting from section 6 we concentrate on two examples in the group _F 4 . These examples 
are typical and represent the general process. Therefore we study them in detail. In the 
notations of the table at the end of section 4, we consider in section 8 the global integral 
attached to the unipotent orbit O = A 2 and in section 10 the integral attached to the 
unipotent integral O = A\. 

As mentioned above, this paper is a starting point towards a certain systematic study of 
constructing global integrals, first in the exceptional groups, and later in classical groups. 
Clearly there are many details which need to be addressed. First, in this project we assume 
that 7r is generic. However, one can study also the cases where the representation it is 
cuspidal and not generic. Secondly, the integrals which we construct here are natural in a 
certain sense. But one can also study global integrals, which satisfy the dimension identity 
given in section 4, but not of the type which we construct in this paper. Finally, there is the 
interesting question of determining which L functions we obtain for those integrals which 
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are Eulerian. This is, of course, a local question, and needs to be addressed by means of 
local computations. In the two examples we consider in this paper, we obtain two examples 
of such integrals which are new. We hope to study these topics in the future. 

2. Basic notations 

Let H denote one of the exceptional groups F4 or Ei for i = 6,7, 8. By this we mean the 
unique split connected simply connected simple algebraic group of the given type defined 
over our number field F. We shall label the simple roots of H as follows. First, for the group 
H = F4, the Dynkin is given by 

ct\ 0/.2 c*3 Q4 

==>== 

As for the group H = E s , we label the simple roots as 

d\ Q13 0:4 a$ dQ CK7 ag 





«2 

The Dynkin diagram for the groups H = E e and H = E 7 , are the ones derived from the 
diagram for Es omitting the relevant roots. 

For construction of several of our integrals, we will need to use the similitude groups for 
E 6 and E 7 . We shall denote these groups by GE Q and GE 7 , and use the notations defined 
in [G2] . 

We assume H to be equipped with a choice of maximal torus T and Borel subgroup, 
B = TU^ X . Here is the unipotent radical of B, which is a maximal unipotent subgroup 
of H. If G is any T-stable subgroup of H we denote the set of roots of T in G by $(G, T). 
We also assume H to be equipped with a realization, in the sense of SprJ , i.e., a choice 
of isomorphism x a : G a — > U a for each root a G Q(H,T), subject to certain compatibility 
conditions. Here U a is the one- dimensional subgroup of H attached to a. We further assume 
that the structure constants for this realization are given as in [G-S] . 

For each root a, the groups U a and U- a generate a subgroup of H which is isomorphic to 
SL 2 . We denote this subgroup by SL^. Likewise, given two roots a and (3, the subgroup of 
H generated by U± a and U±p, is denoted by SL^ 01 '^ if it is isomorphic to SL 3 , by Sp\ a '^ if 
it is isomorphic to Sp^, etc. 

We shall denote the maximal parabolic subgroups of H as follows. Let P be a standard 
maximal parabolic subgroup of H . Let denote the unique simple root of H such that the 
one parameter unipotent subgroup x ai (r) is in the unipotent radical of P. We shall then 
denote P by Pi. We also denote the unipotent radical by U{ and the standard Levi subgroup 
by Mi. 
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The choice of realization allows us to work with elements of H via explicit generators and 
relations. In this approach we shall refer to specific roots of H very often. If ax, . . . , a r are 
the simple roots of H, we identify the root Yli=i n i a % with the tuple (n 1; . . . , n r ). Since none 
of the coefficients n« ever exceeds 9, we shall normally simply run the digits together. For 
example, the root 2a\ + 3a 2 + 4ct3 + 2a 4 of F 4 will be denoted 2342 or (2342). 

2.0.1. Some conventions for defining characters. Suppose that U is a unipotent group and 
we need to define a character of U. This will occur very frequently in this work. We introduce 
two convenient notational conventions for this, at least in the case when U is T-stable, in 
which case it is a product of groups U a . Assume that this is so. Fix a total ordering on 
$([/, T). Then we may define an isomorphism of varieties Gf imf/ — > U by 

(Ua)aE$(U,T) ^ ] J X a (u a ), 
a&b{U,T) 

with the product defined using our fixed total ordering. This in effect defines "coordinates" 
(u a )ae^(u,T) on U. Further, the a-coordinate u a of u G U depends on the choice of total 
ordering only if U a lies in the commutator subgroup of U. In particular, characters may be 
written using these coordinates without ambiguity, even when the total ordering has not 
been specified. For example, one can define a character of the maximal unipotent subgroup 

of F 4 by ipu maK ( u ) = ^(^oioo + ^ooio + moooi)- Note that Mi 00 o, ^oioo, u oow and w 00 oi are tne onl y 
coordinates of u G U mSiX which are well defined, independently of the total ordering chosen. 
On the other hand, they are also the only coordinates on which a character of U max may 
depend. Finally, note that while these coordinates are independent of the choice of total 
order, they are very much dependent on the choice of realization {x a : a G <&(H, T)}. But 
the realization has been fixed once and for all, so this is no cause for concern. 
Sometimes a different convention is more convenient. If we write 

^u{xp 1 {ri)xp 2 {r 2 )u) = ip(n + r 2 ), 

or the like, the convention is that the restriction of U to U a is trivial for all a G <&(U,T) 
except for those listed (here (3i, and /3 2 ). Also u' is an element of the product of the groups 
U a , a G $(£/, T) with the listed roots excluded. 

3. Construction of Fourier Coefficients 

In this section we shall explain how to construct a Fourier coefficient from a weighted dia- 
gram corresponding to a unipotent orbit. Our construction is essentially a slight refinement 

of the one given in |G-R-Slj . and applies to any split connected reductive algebraic group. 
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Our main interest are constructions in the exceptional groups, however this setup holds for 
classical groups as well. Let H denote a split connected reductive algebraic group. 

We shall work in characteristic zero. Consequently, we may pass freely back and forth 
between unipotent orbits in a reductive group and nilpotent orbits in its Lie algebra. Indeed, 
the exponential map gives an equivariant isomorphism between the nilpotent and unipotent 
subvarieties. We may also pass back and forth between adjoint orbits in the Lie algebra and 
coadjoint orbits in its dual, as described in |C-M] . section 1.3. 

We use the classification results described in |C], particularly the tables of orbits which 
begin on page 401, as well as the Bala-Carter labels used therein. For the list of dimensions 
of these orbits, we use |C-M] page 128. A well known result of Dynkin which is described 
in section 5.6 of fC] associates with any unipotent orbit a Dynkin diagram whose roots are 
labeled with zeros, ones and twos, which determines the orbit completely. For exceptional 
groups, the Bala-Carter label or the weighted Dynkin diagram is the most common method 
of specifying an orbit. For classical groups it is more common to use partitions as in |G-R-Slj . 
However, the parametrization via weighted Dynkin diagrams is valid in general and, indeed, 
the first part of the construction given in |G-R-Slj amounts to recovering the weighted 
Dynkin diagram attached to an orbit from the partition. 

Let O denote a unipotent orbit for the group H. (We shall identify each algebraic group 
which we consider with its F points where F is a fixed algebraic closure of F.) First, we shall 
associate to O a parabolic subgroup of H which we shall denote by Pq. We shall write its 
Levi decomposition P G = M Uq- Here M a is the standard Levi subgroup of Pq and Uq is 
its unipotent radical. 

The parabolic subgroup Pq is defined to be the standard parabolic subgroup of H whose 
Levi part is generated by the maximal torus of H, and all copies of SL 2 corresponding to 
those simple roots which are labeled by zero in the diagram. For example, in the group 
H = E 6 the diagram corresponding to the unipotent class O = 2A 2 is given by 



(2) | 



Here and throughout, roots without a number are considered as labeled with the number 
zero. In other words, the roots a 2 , «3, «4 and a$ are labeled with the number zero. Thus, in 
this case M G = Spin 8 -GL\. 

As a second example, consider in H = F 4 the unipotent class O = B 2 . Its diagram is 
given by 

==>== 
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Therefore, the Levi part of Pq is given by Mq = Sp^ ■ GL\. 

Since Pq is standard, Uq is generated by one dimensional unipotent subgroups x a (r), 
where a is positive. For example, in the case of diagram (J2J), the unipotent group Uq is 
generated by all x a (r) such that if we write a = Y^i=i n i a ^ then either n\ > or n 6 > 0. 
Thus dim U = 48. 

Let i/j denote a nontrivial character of F\A. To the unipotent orbit O we shall associate a 
character ip Uo defined on a subgroup of Uq. In the case of the classical group this procedure 
is explained in detail in |G1] section two. For the exceptional groups this is done in a similar 
way. To describe the construction, we fix some notations. 

We will say that the root a is in Uq if the one parameter unipotent subgroup x a (r) is 
in Uq- Assume that the simple roots oc^ , . . . , ai r are the simple roots which are labeled 
zero in the diagram corresponding to the unipotent orbit O. By definition, a positive root 
a = nj«j is in Uq if and only if n ir + . . . + n ir > 0. For each k > we set Uq equal to 
the product of the groups U a for a in the set {a = J2i n i a i '■ + . . . + n ir > k}. It is 
easy to see that U { q ] is a group and that L { $ = ugt /ug +1) is an abelian group. We also 
define Vq^ as the product of the groups U a corresponding to those roots a = rijaj with 
°i n i — Here, q G {0, 1, 2} is the weight attached to the simple root a« in the weighted 
Dynkin diagram of O. We consider the group Vg'/Vg*. It may equal L a , or L a , or some 
combination of the two. The group Mq acts on this group with an open orbit. We say that 
an element of Lq is "in general position" if it is in this orbit. See proposition 5.7.3 of [C] 
for details. It follows from proposition 5.5.9, p. 156 of [C] that the identity component of 
the stabilizer inside Mq of a representative of the open orbit is a reductive group. 

To define the character ipu i assume first that the diagram attached to O has only zeros 
and twos. In this case, V^f 1 /Vq^ = and ipu a will be a character of Uq itself. We consider 
the action of Ma on the group Lq . This action is essentially a rational representation of 
Mq and the F-points of the dual rational representation are identified with the set of all 
characters of Uq by our choice of As explained above, over an algebraically closed field, 
the action of Mq on Lq has an open orbit, and the identity component of the stabilizer of 
any element of this orbit is reductive. The same is true of the action of Mq on the dual 
rational representation, which we denote Lq*. However, the action of Mq(F) on L^q'*(F) 
need not have an open orbit. Indeed, the intersection of Lq'*(F) with the open orbit in 
L^q may be a union of several Mq(F) orbits, with stabilizers which are isomorphic to one 
another only over the closure and not over F. This issue is discussed in |Glj in example 2 
page 328. For ipu we choose a character which is in the open orbit. Following the tables 
given in [C] page 401, we shall denote the connected component of the stabilizer by C . 
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Let ip denote an automorphic form defined on the group H(A), It follows from the above 
discussion that the following Fourier coefficient 

(3) ^ Uo ) {g):= J y( ug )^ Uo (u)du (geC(A)) 

U (F)\U (A) 

defines a left-C(F)-invariant function C(A) — > C. 

Lemma 4. The function <p( u ortu ) j s sm ooth, and ip( u o>$u ) anc [ a \\ Q f derivatives have 
moderate growth. 

Proof. We define "smooth" and "moderate growth" as in [MWj . section 1.2.5. We are given 
that <p is smooth. Since (p^ u °>^ u o^ j s obtained by integrating ip over a compact set against a 
smooth, bounded function, it is clear that ^ u °'^ v o"> inherits this property from (p. 

We are also given that tp is of moderate growth. This is defined using a choice of embedding 
H <^-> GLn for some N (though the condition obtained is independent of the choice of 
embedding). If we then define "moderate growth for C using the embedding C H ^ 
GLjy, then it is clear that p^ u °^ u o"> inherits this property as well. □ 

We remark that ^o^uq) a j so inherits the i^-finiteness property of the automorphic form 
(p, provided that the maximal compact subgroups of H^Foo) and C(F 0Q ) are chosen compat- 
ibly, but that it will not , in general be 3-fmite, and therefore that it will not, in general, be 
an automorphic form. (Here, 3 indicates the center of the universal enveloping algebra of 
Lie(C(F 00 )).) We elaborate on this remark slightly in an appendix. 

We consider a few examples. First, let H = E e and O = 2A 2 . The roots in Lq are all 
those roots a = Yl^=i n i a i suc h that n\ + n 6 = 1. There are 16 such roots. The group 
Mq = Sphig -GL\ acts on this group. The representation is reducible, and up to the action 
of GL\ it is equal to two of the three eight dimensional representations of Spin 8 . It follows 
from [C] that the stabilizer of the open orbit is the exceptional group For u e Uq define 

ipUo( u ) = ^1/0(^111100(^1)^101110(^2)^010111(^3)^001111(^4)^') = ip{n + r 2 + r 3 + r 4 ) 

where v! G Uq is any product of unipotent elements x a (r) in Uq such that a is not one of 
the above four roots. Then the stabilizer inside Mo is the exceptional group G 2 . 

As an another example, consider in E 8 , the unipotent class O = D4. Its diagram is given 

by 

— — — — — d — d 
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In this case Lq consists of all roots a = Yl^=i n i a i such that n 7 + n 8 = 1. There are 28 
such roots. The action of Mo = Eq ■ GL\ on the roots in Lq gives a sum of two irreducible 
representations. First, on x aa (r) we obtain a one dimensional representation of Mo- The 
second representation corresponds to the 27 roots a = Ylt=i n * a « such that n 7 = 1 and 
n§ = 0, is the 27 dimensional representation of If we define the character 

1pU a ( U ) = ^(^00000001 (n)a;ii221110(^2)a;ill22110(^3)^01122210(^4)M / ) = + r 2 + r 3 + f 4 ) 

then the stabilizer in Mo is the exceptional group F4. 

As a final example of this type, suppose that the diagram attached to O contains only 
twos and no zeros. Then Mo is the torus of the group H and Uq is the maximal unipotent 
subgroup of H . Therefore, in this case we may take ipu a the standard generic character 
corresponding to our chosen realization. In other words, if o.i are the simple roots of H, then 

i)Uoi u ) = ^Uo^aAn) ■ ■■X am (r m )u') = V>( r l H ^ r m)- 



Next, we consider the case when the diagram attached to the unipotent orbit contains 

-(2: 
o 



(2) 

only zeros and ones. In this case we define the character on the unipotent subgroup U} 



(2) 

More precisely, we consider the action of Mo on the group L , and we define the character 
ip TT (2) in a similar way as we defined the character ipu in the previous case. 

As a first example we consider in the group H = Eg the unipotent orbit O — A±. The 
diagram which corresponds to this unipotent class is 

— — 0. 

(5) I 


1 

(2) 

In this case, the group Uo is a Heisenberg group, and the group L ' has only one root, 
which is (122321). Therefore, the character we attach to this unipotent class is given by 
VVr( 2 ) (^122321 (r)) = V'( r )- m this case, the connected component of the stabilizer C is a group 
of type A 5 . As in integral ([3]), given an automorphic representation a of E e (A), we can define 
the Fourier coefficient which corresponds to this orbit by 

(6) J ^(ugtyyw^du. 

ug\F)\U ( ?\A) 

This Fourier coefficient defines an automorphic function defined on the group C(A). 
As an another example, consider the unipotent class O = Ai in H = F 4 . Its diagram is 

(7) 0==>==0 0. 
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In this case the group U 2 ' consists of the seven roots a = Yut=i n i a i * n H sucn n 4 = 2. If 
we define the character 

i> n w(u) = il> IT (?){x 12 32{r)u') = if)(r), 

then the connected component of the stabilizer inside Mo is the group Spin 6 . 

Returning to the general case of diagrams which contains zeros and ones only, it will be 
convenient to extend the unipotent group we integrate over. This extension, which involves 
the theta representation defined on the double cover of a suitable symplectic group, is dis- 
cussed in [G-R-Slj section 1 in detail for unipotent orbits of the symplectic group. In the 
exceptional groups it is exactly the same. The group Uq/Uq^ has the structure of a general- 
ized Heisenberg group. (By this we mean a group which has a projection onto a Heisenberg 
group such that the kernel is contained in the center. See section [8] for a more detailed defi- 
nition.) This is a general phenomenon, which we explain briefly in section [HJ For example, in 

(3) 

the case of the unipotent class A 1 in E 6 , the group U is trivial, and Uo is the Heisenberg 
group with 21 variables. The action of the group C on Uo by conjugation therefore induces 
a homomorphism into certain symplectic group. In the above example, it is the group Sp 2 o- 
In general there is a projection map I : Uojuf H m where U m is a Heisenberg group 
with m variables. Here m = dimLy + 1. As an example to this phenomena, consider in 
H = F 4 , the unipotent class O = A\ + A±. Its diagram is given by 

==>== 

In this case, the roots in Lq are all twelve roots a = Ylt=i n i a i such that n<i = 1, and the 
roots in L are the six roots with ri2 = 2. The group Uo/U is a generalized Heisenberg 
group, and in this case we define / : Uo/U^ — > f-Li 3 as follows. Recall that Hi 3 is G^xG^x G a 
equipped with the operation 



(xi\yi\zi) ■ (a^M^) = [x 1 +x 2 



Vi + 2/2 



zi + z 2 + -(aci ■ t yi - V\ ■ tx 2 ) 



Here xi,yx 2 ,y 2 G G^, realized as row vectors, and t denotes the lower transpose: 




\X\, 

Also z\,z 2 G G a . (We separate the components of ~Hi3 with vertical bars to aid legibility in 
computations where individual entries in the row vectors are written out.) 

The mapping of onto the center of should be an element of L$'* (the Mo-module 
dual to Lq ) in general position. One option is as follows. For any u' G U a we have 

/(a:i22o(n)a;i22i(r2)a;i222(r3)a;i23i(^4)a;i232(r5)a;i242(r6)M / ) = (0|0|r 3 + r 4 ) 
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It is not hard to check that the stabilizer inside Mq is a group of type A\ + A\ as indicated 
in the table in [U]. 

In conjunction with the commutator map Lq x Lq — > Lq , the linear form on Lq 
which has been chosen determines a skew-symmetric form on Lq x Lq . To extend it to a 
projection Uq — > H13, one needs an isomorphism Lq — > G„ x G^j such that the preimage of 
{(x|0|0) : x G G^} in Lq is isotropic with respect to this skew-symmetric form, and so is 
the preimage of {((%|0) : y G G^}. 

In the case at hand, one might take 

Kxo W o(m 1 )xu o(m 2 )xoiio (m 3 )x ni0 (m 4 )x m (m 5 )x i 2 o0™ 6 )) = (^|0|0) 
where X = (m 1; rri2, ■ ■ ■ , m§) G G„. Then I will map 

{^ii2o(^i)^Hii(^2)xoi2i(rn, 3 )xii2i(m 4 )a;oi22(rn, 5 )xii22(rn6) ■mi,..., m 6 } 

isomorphically onto {(0|y|0) : Y G G^} C H13. The precise isomorphism will depend on the 
structure constants of F 4 . This issue is discussed in more detail later on in section El 

Returning to the general case, having fixed a projection map / : Uq/U —> T^fc+i we get 
a homomorphism of the group C into Sp2k- This is because Sp2k may be identified with the 
group of automorphisms of %2k+i which restrict to the identity on the center, and because 
the action of C on Uq will be given by such automorphisms. 

Let 6^ denote the theta representation of H2k+i x Sp 2k (A). Then the above discussion 
indicates that the following integral 

8i{l{v)g)Vcr{ug)du 

is well defined and is left invariant under g G C(F). Here Q, is a vector in the space of Of. 
Depending on whether the group C(A) splits inside S , p 2 fc(A), integral (jSJ) is a well defined 
function on either C(A) or the metaplectic double cover C(A). Notice the difference between 
integrals ([6]) and OH]). Clearly the domain of integration is different. But more importantly, 
integral OH]) will sometimes define an automorphic function on the group C(A), and other 
times will define a genuine automorphic function on the metaplectic double cover of C(A), 
whereas integral ([H]) always defines an automorphic function of C(A). As we will work only 
with integrals of the type OH]), this will be important for our construction. 

The final case to consider is the case when the diagram which corresponds to the given 
unipotent orbit O consists of zeros, ones and twos. In this case we combine the above two 



I 



(2) (3) 

cases. In order to explain this, recall the group 

Vo l V o • In the 

case when the weighted 



diagram has only zeros and twos, it is Lq . In the case when there are only zeros and ones, it is 
(2) 

Lq. If both ones and twos are present, it contains a part of each of these groups. The group 
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U /V® is the direct product of the abelian group (V^/V* ) H 4\ aud a generalized 
Heisenberg group with center (Vq /Vq ) fl L D . Define projections I : Uq —> H271+1 and 
V :U ^ (V$ 2) /v£ 3) ) n L^. Here 2n is the dimension of Vq 1] /V§ ] . 

As in [G-R-Sl] equation (1.3), the corresponding Fourier coefficient is then given by 

(9) J e^{l{u)g) Va {ug)^ Uo {l'{u))du 

U (F)\U (A) 

Here 9^ is a vector in the theta representation defined on the suitable symplectic group. As 

(2) (2) 

before we have a character of Vq (F)\V (A), which, in this case is a combination of ipu Q 
and the central character of 9^, defined on the group (V^ /V^) fl Lq using the projection 

(2) (3) 

I. This character corresponds to an element of the Mo-module dual to Vq /Vq , and must 
be in the open orbit for the Mo-action. 

As an example, consider in H = F4 the unipotent class O = B2. Its diagram is given by 

==>== 

As was mentioned above, the Levi part of Pq is given by Mq = Sp^ ■ GL\. In this case, 
the action of Mq on Lq decomposes as a direct sum of two irreducible subrepresentations. 
One subrepresentation corresponds to the subgroup of Mq on which T acts with the roots 
(1000), (1100), (1110), (1120), (1220), and the other corresponds to the subgroup of Mq on 

which T acts with the roots (0001), (0011), (0111), (0121). 

(2) 

Similarly the action of Mq on L Q decomposes as a direct sum of two irreducible subrepre- 
sentations. One corresponds to a four- dimensional subgroup with roots, (1111), (1121), (1221), 
(1231), and the other corresponds to the subgroup £/oi22- The function J2i n i a i ^ ^2i n i c i 
in this case is Ylt=i n i a i ^ ^1 + n 4 . The group V^f 1 /V^ 1 consists of the first component 
of Lq and the second component of Lq . In other words, it corresponds to the six roots 
(1000), (1100), (1110), (1120), (1220) and (0122). The group U /V§ ] is the product of a five 
dimensional abelian group corresponding to the roots (1000), (1100), (1110), (1120), (1220) 
and a five dimensional Heisenberg group corresponding to the roots (0001), (0011), (0111), (0121), 
and (0122). 

In order to define a Fourier coefficient in this case, we need to take a character in general 
position on the five dimensional abelian group, as well as a nontrivial character of the center 
of the Heisenberg group. We may then extend the latter to a theta representation. 

Our five dimensional representation of Mq can be thought of as the standard representa- 
tion of (with Sp4 then appearing in its guise as Spin 5 ). A point in this representation 
is in general position if the Mo-invariant quadratic form does not vanish at that point. The 
character ipu {u) = V'(^iiio) is easily seen to be in general position. The identity component 
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of its stabilizer in Spin 5 is Spin 4 = SL 2 X SL 2 . The stabilizer in Mq contains an additional 
one-dimensional torus, however, if we restrict to elements of Mq which also act trivially on 
^0122, the resulting group is SL 2 2 ■ SL® 120 , and indeed, the tables in |U| reflect that for this 
unipotent class, the group C is of type A\ x A±. As mentioned earlier, the root subgroups 
U a for the roots (0001), (0011), (0111), (0121), and (0122) form a subgroup isomorphic to 
Thus, the corresponding Fourier coefficient, for this unipotent class is given by integral 
OU). Here 9^ is a vector in the space of 0f , the theta representation defined on Sp 4 (A) } and 
I is the projection from Uq to H5. As in previous examples, the character VVrW which is 
nontrivial on the group Uqi 22 is built in the theta function. We should mention, that since 
the group SL 2 X SL 2 does not split under the cover of Sp 4 (A), it follows that the above 
integral defines a genuine automorphic function on the metaplectic cover SL 2 (A) x SL 2 (A). 
There is a third type of Fourier coefficient which can be attached to an orbit such that the 

(2) 

weighted Dynkin diagram has ones in it in certain cases. To be specific: recall that Uq/V 
is essentially a symplectic vector space, equipped with a nondegenerate skew-symmetric form 

(3/2*) 

which is fixed by the group C. Let W be a maximal isotropic subspace, and let U denote 

(3/2) 

the preimage of W in Uq. Let ip TT (3/2) denote the trivial extension of ip TT (2) to U a . Then 
the third type of Fourier coefficient is 

(10) J ^ a {ug)^ v (zi2) du. 

U^ /2) {F)\U^ /2) (k) 

Note that this integral defines a function on C(F)\C(A), only if C normalizes Uq , i.e., if 

(2) 

the maximal isotropic subspace W C Uq/Vq is C-invariant. 

A coefficient of this type was used in |B-G] . Also, the integral given in |J-S2] for the 
convolution L-function of GL n x GL m involves a Fourier coefficient of this type whenever 
n — m is positive and even. 

For completeness, we record the analogue of lemma 1.1 of |G-R-S1] . 

Lemma 11. Let / : 'H 2n+ i(F)\H 2n+ i(A) — > C be any smooth function. The following are 
equivalent: 



f((0\0\z)u)ij(z) dz = 0, (Vu E n 2n+1 (A)), 



f((0\y\z)u)iP(z) dydz = 0, (Vu G H 2n+1 (A)) 

F\A (F\A) n 
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f(u'u)e^(u') dv! = 0, (Vw G H 2n+1 {A), G S(A n )). 

H2n + l{F)\H2n + l(^) 

Proof. The same arguments given to prove lemma 1.1 of [G-R-Sl] actually prove this stronger 
statement. □ 

Corollary 12. For an automorphic representation (cr, V a ) of H(A) (or a covering group), 
the following are equivalent: 

1) : integral §6§ is zero for all ip a G V ai 

2) : integral is zero for all (f a G V a and all G S(A n ), 

3) : integral (TTOT) is zero for all G V a . 

4. Constructing Global Integrals 

From the discussion in the previous section, it follows that given a unipotent orbit O of an 
exceptional group H, we can associate with it a set of Fourier coefficients, which are defined 
on any automorphic representation of the group H(A). In this section we shall use this to 
construct certain global integrals. 

We start with the case where the corresponding diagram has only zeros and twos. Let 
7r denote a cuspidal representation defined on the group C(A). Given a maximal parabolic 
subgroup P of H, with Levi decomposition P = MU, we let E T (h, s) denote an Eisenstein 
series of the group H(A) formed using an element of the induced representation Indp^ t5 s p . 
Here r is any automorphic representation of the group M(A). 

The global integral we consider in this case is given by 



(13) J j (p w (g)E T (ug,s)il) Uo (u)dudg 

C(F)\C(A) Uo(F)\UoW 

In the case when the diagram corresponding to the unipotent orbit contains ones, we 
consider the integral 

(14) J j p w (g)ep(u)g)E T (ug,s)-iP Uo (l'(u))dudg 

C(F)\C(A) Uo(F)\UoW 

where I and /' project Uq/Vq onto its Heisenberg and abelian factors respectively. (See 
the discussion before equation Q.) As explained in the previous section, in this type of 
integral, the Fourier coefficient given by the integration along Uo{F)\Uo(A) may define a 
genuine automorphic function on (7(A), the double cover of (7(A). If this is the case, then 
for integral (JHJ) to be well defined, one of the representations n or E T (g, s) should be defined 
on the double cover of the relevant group. 
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Both integrals (|T3|) and (|T4|) converge absolutely whenever s is not a pole of the Eisenstein 
series. This follows from the moderate growth property of automorphic forms and their 
Fourier coefficients, and the rapid decrease property of cusp forms. Also, each defines a 
meromorphic function on the whole complex plane, simply because the same is true of the 
Eisenstein series. 

There are two other families of global integrals which are of interest. We refer to as 
reductive type integrals, and formally attach them to the zero nilpotent orbit. (For which 
C equals H itself.) The first reductive type are integrals of the form 

(15) J (p n (h)e(h)E T (h,s)dh 

H(F)\H(A) 

where 6(h) is an element of some irreducible automorphic representation 6 of H(A). Here 
7r denotes a cuspidal representation of the group if (A). (The similarity in notation between 
and is deliberate: in practice G is usually attached to a small orbit, and in this way is 
like G^, which is attached to the minimal orbit of Sp2n.) The second family will be integrals 
of the type 




<p n (g)E T (g,s) dg 



Z(A)G(F)\G(A) 

where G is a reductive subgroup in H, Z is its center, and tt is a cuspidal representation 
defined on G(A). 

As mentioned before, when we deal with the groups E& and Ej, we may need to use their 
similitude groups, in order to guarantee that the integral will unfold to a suitable model. 
In these cases, we may define the group C as the stabilizer of a suitable character ipu a in 
the similitude group. It will certainly contain the center, Z of the similitude group, and we 
replace the integral over C(F)\C(A) by an integral over Z(A)C(F)\C(A). Each of the above 
integrals converges absolutely, and satisfies a functional equation which is derived from the 
Eisenstein series. 

Our main goal is to determine when these integrals are Eulerian. To try to answer this 
question, at least partly, we shall fix some notation. We need a good way to parameterize 
automorphic representations. One way to do it, which is suitable for our problem, is by 
means of Fourier coefficients. 

In [Glj . it is explained how to associate with each unipotent orbit a set of Fourier coef- 
ficients. This was essentially repeated in the previous section. The following definition is a 
way to attach to any automorphic representation a set of unipotent orbits. 

16 



Definition 17. ( |G1] Definition 2.1) Let a denote an automorphic representation of a given 
split reductive group G(A), or of a metaplectic cover of such a group. We shall denote by 
Orba(cr) the set of all unipotent orbits O of G such that a has a nonzero Fourier coefficient 
which corresponds to the unipotent orbit O, and by Og{&) the set of maximal elements of 
Orb G (a). 

The structure of the set Oq{o) is not yet well understood. In |Glj there are several 
conjectures and assertions which are related to this set. To each representation which appears 
in integrals f[T3"j) . ( I14p . (fT5|) and f[T6"j) we shall attach a number which we refer to as the 
Gelfand-Kirillov dimension of the representation. (By way of analogy with the "dimension" 
introduced in jK], theorem 2.4.1, (iib). Cf. also the references on p. 158 of [K] for related 
results on real Lie groups and the connection with the notion originally introduced by Gelfand 
and Kirillov.) 

Definition 18. (1) Assume that the set Om{t) contains a single unipotent orbit O. Let 
E T (-,s) denote the Eisenstein series defined right before integral f jl~3|) . Then define 
the Gelfand-Kirillov dimension dim r of r to be |dim O. For Re s large define the 
Gelfand-Kirillov dimension of E T (-,s) to be dim U + dim r. 

(2) Since we will assume that tt is generic, then we define dim n = dim U^ ax where U^ ax 
is the maximal unipotent subgroup of C. 

(3) For the theta representation 0^ defined on H2n.+i (A) x Sp 2n (A) we define dim 6^ = n. 

(4) Finally, for integrals of the type ffl~5l) . we assume that Oh(Q) is a singleton set {O}, 
and define dim = | dim O. 

Remark: Observe that Oo(tt) is the singleton set containing the maximal orbit of C, 
which has dimension 2dimf/^ ax . Also Os P2n {®t) ^ s the singleton set containing the minimal 
orbit of Sp2n, which has dimension 2n. Finally, suppose that £ T)S is the space of Eisenstein 
series attached to the induced representation Indp^y t5 s p . Then, following |Glj . proposition 
5.16, we expect that 

O m (t) = {£>} O h {8 t ) = {£>'}, where dimC = dimC + 2dimC/(P). 

Actually, we can prove it for low rank groups, including the exceptional group F4. However, 
since we will not need it, we omit it. 

Note that the representation r, or the representation 6 appearing in (fT5]l could itself 
be an Eisenstein series. In this case the dimension is defined using the unique element 
of Om{t) or Ojj{®), and not using definition [T71 The motivation for this is that in the 
practice of unfolding one needs to know information regarding what Fourier coefficients 
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these representations support, and prefers, where possible, to make arguments that are valid 
for any representation supporting a given set of Fourier coefficients. 

Next we consider the cuspidal representation ir which appears in integrals (Til?]) and (JHJ). 
Even though our main interest is the case when 7r is generic, and hence dim 7r is well defined, 
for possible future applications we prove 

Lemma 19. Assume that H = F4 or H = Eg. For any nontrivial unipotent orbit of H, let 
7r be an irreducible cuspidal representation of C(A). Then dim7r is well defined. 

Proof. The lemma is proved by a case by case consideration. The type of C for each orbit is 
given in the tables of [Cj. It will be a group of type Ai, A 2 , C2, G2, A 3 , B 3 , C3, A 5 , or some 
combination of these. 

If G is a simple group of any of the types listed above, except for C3 and A$, then the 
orbits of G are totally ordered, and so every subset has a unique maximal element. For C3, 
the set of all special orbits is totally ordered, and it is proved in |G-R-Slj theorem 2.1 that 
any orbit appearing in Ocijt) must be special. (For a definition of "special," see [Cj, p. 389. 
For an alternate description for classical groups, see |C-Mj . p. 100.) 

This leaves a single orbit in Eq when the stabilizer is of type A5. In fact (since we work 
with the split form of Eq) the stabilizer for this case is isomorphic to SLq (If we work in 
GE 6 , the stabilizer is isomorphic to {g G GL Q : det g is a square}. This case was considered 
in |G-Hlj .) Every cuspidal representation of SL& is generic with respect to some generic 
character, so in this case dim ir is always defined and equal to the dimension of the maximal 
unipotent subgroup in SL e . □ 

Finally, we consider the representations occur in integrals ffl5|) and (fT6l) . In these cases, 
we will assume that ir is generic, and for the function 9 which occurs in integral (fT5|) . we 
will assume that its dimension is well defined. This will not cause a problem, since the for 
cases we will consider this assumption will be satisfied automatically. We record all this in 
the next lemma. 

Lemma 20. With the above assumptions, the notion of dimension is well defined for any 
representation occurring in integrals f|T3|) . (JHj), ([15]) and f|T6|) . 

Recall that our goal is to study the question when are the integrals ffTBl - ffTBl) are Eulerian. 
Following the discussion in the introduction of [G3j we introduce the following definition. 

Definition 21. (The basic equation:) An integral of the type (TT31) . (fill) . (Flo]) , or (|T5|) . 
is said to satisfy the basic equation if the sum of the Gelfand-Kirillov dimensions of the 
representations involved in the integral is equal to the dimension of the domain over which 
we integrate. 
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We now write the basic equation in detail. Consider first integral (|T3|) . The basic equation 
in this case is given by 

(22) dimC + dimU a = dimvr + dim E T (-,s) 
For integral ( I14p the equation is given by 

dimC + dim Uq = dini7r + dimE T (-, s) + dim <df. 

1 (2) 

Here k = ^(dimUo — dim Vq ) is the unique integer such that Uq has a projection onto 

(2) 

H2/C+1, with the kernel of this projection contained in Vq . In the first identity above, we 
have dim Uq = \ dim O, while in the second case, we have dim Uq — dim = | dim O. This 
follows from lemma 4.1.1, p. 56 of |C-M] . 

Hence, it follows that if we start from a nonzero unipotent orbit O then the relevant 
integral f|T3|) or f fT4|) satisfies the basic equation if and only if it satisfies the equation 

dimC + - dim O = dini7r + dimE T (-, s) 

Since we assumed that it is generic, it follows that dim 7r = dim Uq where Uq is the maximal 
unipotent subgroup of C. From definition [TBI we also have that dim E T (-,s) = dimU(P) + 
dimr. Plugging this inside the above equation we obtain the basic equation 

(23) dimr = dim B c + - dim O - dim U(P) 

where Be is the Borel subgroup of C. Notice that this last formula holds also for integral 
f|T6|) . if we take dim O = and C = G. Arguing in a similar way for integral f|T5|) . we deduce 
that the formula that should hold in this case is 

(24) dim 8 + dimr = dim B H - dim U(P) 

where Bh is the Borel subgroup of H. 

We consider a few examples. We start with an example related to integral f|T5|) for the 
group H = F4. In this case, equation fl2"4"]) reduces to dim^ + dimr = 28 — dimU(P). Since 
F4 has two maximal parabolic subgroups P such that dim U (P) = 20, and two such that 
dim U(P) = 15, it follows that there are two possibilities. First suppose that dim U (P) = 20. 
This is the case if P = P2 or P3. Then ( 1241) reduces to dim# + dimr = 8. Now, dim# is 
equal to one half of the dimension of a nontrivial nilpotent orbit of F4. The dimensions of 
the orbits are listed on page 128 of |C-Mj . The minimal orbit has dimension 16, so it follows 
that in this case 9 must be attached to the minimal orbit, and r must be a character. An 
example of a representation which is attached to the minimal orbit of _F 4 is constructed and 
studied in [G4j . In fact, it is a representation which is defined on the double cover of F 4 (A). 

Now suppose that dimU(P) = 15. This is the case when P = Pi or P = P 4 , so that 
the Levi part is isomorphic to GSp^ or GSpin 7 , respectively. In this case, (1241) reduces to 
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dim# + dimr = 13. It then follows from |C-M] page 128, that either 9 is attached to the 
minimal orbit, and dim# = 8, or else 9 is attached to the unipotent orbit A\, and dim 6* = 11. 
If dim 9 is 11, then we need dim r = 2. However, neither GSp^ nor GSpin 7 , has an orbit whose 
dimension is 4. So, once again 9 must be attached to the minimal orbit. The basic equation 
will be satisfied if we can find a representation r defined on GSpe or GSpin 7 , such that the 
dimension is five. Each of these two groups has a unique unipotent orbit O of dimension 
10. The unique 10-dimensional unipotent orbit of GSp§ is (2 2 1 2 ), and that of GSpin 7 is 
(31 5 ). (For the parametrization of orbits by partitions, see |C-M] . p. 70. To compute the 
dimension, use |C-M] . §5.2, and lemma 4.1.1.) 

From this, it follows that there are essentially six different possibilities for an integral of 
the form f lT5|) . First, suppose that the Eisenstein series E T (h, s) comes from a representation 
induced from either Pi or P4, and r is any representation of GSp§ or GSpin 7 whose dimension 
is five. Since the representation 9 is on the double cover, it follows that either 7r or E T (g, s) 
has to be defined on the double cover. If we form the Eisenstein series on the double cover 
of ff , we obtain an integral of the form 

J ip n (h)9(h)E T (h,s)dh. 

H(F)\H(A) 

Here r is an automorphic representation defined on the double cover of GSps(A) or GSpin 7 (A) 
whose dimension is five. A second option is to let 7r be a cuspidal representation defined on 
the metaplectic double cover if (A) of if (A). Thus, in this case, we have 

J ^{h)9{h)E T (h,s)dh. 

H(F)\H(A) 

In this option r is an automorphic representation of GSpe(A) or GSpin 7 (A) whose dimension 
is five. 

Now suppose that the Eisenstein series E T (h,s) is attached to the parabolic subgroup 
P2 or P 3 . In this case r must be a character, and it follows that E T (h,s) can not be a 
genuine function on the double cover of if (A). Hence we must assume that tt is a cuspidal 
representation defined on the double cover of if (A). The integral we construct is similar to 
the above, where the only change is the choice of the Eisenstein series. 

As a second example, consider the unipotent orbit O — A\ for the group if = E§. 

Its diagram is given in fl5]). The relevant equation to consider is ( 123]) . It follows from 

[G-M] that |dimC = 11, and from [Q page 402, it follows that C is of type A 5 . Thus 

dim Be = 20. Hence equation (T231 is dimr = 31 — dimU(P). The group E$ has 4 non 

associated maximal standard parabolic subgroups. Their unipotent radicals have dimensions 

dim C/(Pi) = 16,dim£/(P 2 ) = 21, dim C/(P 3 ) = 25 and dimC/(P 4 ) = 29, respectively. The 
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integral to consider in this case is the one written in ( 1141) . We consider each parabolic in 
turn. First, if P = P\, then dimr = 15. The Levi part of Pi is a group of type D 5 . It is not 
hard to check that this group has a unique unipotent orbit such that half of its dimension 
is 15. This unipotent orbit is (3 3 1). Next, when P = P2, we obtain dimr = 10. Since 
the Levi part of P2 is a group of type A 5 , and since this group has no unipotent orbit of 
dimension twenty, it follows that the set of solutions to equation fl23J) is empty in this case. 
For P = P 3 we obtain dimr = 6. The Levi part of this parabolic subgroup is a group of type 
A\ x A4. There is one possibility. On the group of type A\ we take a representation attached 
to the orbit (l 2 ), i.e., a character, and, on the group of type A4, we take a representation 
attached to the unipotent orbit (2 2 1). We denote this by (1 2 |2 2 1). Finally, for P = P 4 , we 
have dimr = 2 and since the Levi part of this parabolic subgroup is of type A2 x A\ x A%, 
then the only choice is take a representation attached to the unipotent orbit (21) on one of 
the two /^-components, and take characters on the other two components. By symmetry, 
it does not matter which of the two v4 2 -components we take the representation attached to 
(21) on, so in this case there is only one possibility, which we represent by (21|1 2 |1 3 ). 

In the following table we summarize the following data for the group H = P 4 . First, in the 
leftmost column we label the unipotent orbit in question. The non special orbits are labeled 
with bold letters. In the next column we write the type of the group C. Since H = P 4 , 
we have four maximal parabolic subgroups. In the next two columns we write down the 
dimensions of r required so that equation (12"3"1) holds. Since the unipotent radicals of Pi and 
P4 have the same dimension, we can write them in the same column. Similarly for P 2 and 
P 3 . In the next four columns we write down possible unipotent orbits, defined on the Levi 
part of P, whose dimension is twice the dimension of r. For example, consider the unipotent 
orbit P>2- Since it is not special we label it with bold letters. The stabilizer is of type A1+A1. 
When P is such that its Levi part is of type C3 or P3, then the dimension of r required to 
satisfy equation (|23|) is seven. The group of type C 3 has 2 unipotent orbits of dimension 14. 
They are (3 2 ) and (41 2 ). They are listed at the fifth column. Notice, that since (41 2 ) is not 
special we labeled it with bold letters. 

Finally, the first row in the table is devoted to the integral of type (fl6l) . The numbers 
indicated are the numbers dim r + dim 9 such that equation (|24p is satisfied. For example, if 
P = Pi then dimr + dim^ must equal 13, and, as discussed above, this means that 9 must 
be attached to the minimal orbit of P4. The Bala-Carter label of this orbit is Ai. Also, r 
must be attached to the orbit (2 2 1 2 ). 

One may also consider what we refer as the "dual integrals" to integrals (9) and (10), in 
which the roles of the cusp form and the Eisenstein series are reversed. For example, the 
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integral dual to integral (9) is given by 

cpi r (ug)E T (g, s)^) Uo (u) du dg, 

C(F)\C(A) U (F)\UoW 

where <p n is a cusp form in the space of a cuspidal automorphic representation of H(F)\H(A) 
and E T is an Eisenstein series on C(F)\C(A). Convergence of such integrals follows from 
the fact that the Fourier coefficients of cusp forms inherit their rapid decay property, as has 
recently been established in general by Miller and Schmid |MS] . However, in this paper we 
are mainly interested in integrals such that the representation ir is generic, and which satisfy 
the equation 

dim C + -dim O = dim it + dim E T (-,s) 

Since we require that tt is generic, then dim n = 24. Further, dim E T (-,s) is strictly 
positive. To be precise, it is at least equal to the smallest integer which is the dimension 
of the unipotent radical of a parabolic of C. There are only three nonzero orbits in F 4 such 
that | dim O + dim C — 24 is positive. These are A 1; Ai, and A 2 . In the case of Ai, we have 
| dim O + dim C — 24 = 2, and C = SL4. This implies dim E T (-,s) > 3, ruling out this case. 

For A2 we have | dim O + dimC — 24 = 5, and C = If P is either of the maximal 
parabolic subgroups of G 2 , then the dimension of the unipotent radical of P is 5, and so 
dim £■,-(•, s) = 5 if r is a character of the Levi part of P. One of these cases can be reduced 
to a Whittaker integral, and appears in |G-R] . 

For Ai, we have | dim O + dimC — 24 = 5, and C = Sp^. It is possible to arrange that 
dim. E T (-,s) = 5 only by taking E T (-,s) to be an Eisenstein series attached to the induced 
representations Ind^ 6 ^ t5 s p , where P the Levi subgroup of P is isomorphic to GL 1 x Sp^, 
and t is a character. 

Thus, there are in effect only three cases for this type of integral, and one of them has 
already appeared in the literature. We remark that a preliminary analysis suggests that 
neither of the others can unfold to a Whittaker integral. 
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The group F 4 
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5. Unfolding Global integrals 

In this section we indicate several steps in the unfolding of integrals (fT3|) and (|T^|) . The 
integrals (I15p and ( I16p are treated similarly. Integral (I14p is more complicated since it also 
involves the theta representation. However, at least the first steps are similar, hence we will 
first concentrate on integral (jl~3l) . and then discuss integral (jT4")) more briefly. We will also 
return to integral (jl~4l in section El 

5.1. The General Process. The purpose of the unfolding process is to determine if the 
integral is Eulerian, by reducing things to certain functionals defined on the representations 
in question. Then one studies if these functionals are unique. For example, integral f ll~3]) is 
given by 

(25) 1= J J (p n (g)E T (ug,s)i) Uo (u)dudg 

C(F)\C(A)U (F)\Uo(A) 

The two representations which are of import to us are n and r. After we finish the unfolding 
process we will obtain certain inner integration, which may be an integral over a reductive 
group, or a Fourier coefficient or some combination of the two. Since we assume that the 
cuspidal representation 7r is generic, it is natural to hope to obtain a Whittaker coefficient 
after the unfolding process is finished. We assume that r has Gelfand-Kirillov dimension as 
specified by the dimension formula. In most cases, this means that r is not generic. 

We now write several steps in the unfolding process. Unfolding the Eisenstein series we 
obtain 

E T (ug 1 s)= Y fAlug,s) = 

~/eP(F)\H(F) 

Y Y fr(wiug,s) 

w£P(F)\H(F)/P {F) r ^(w^P(F)wf]P (F))\P (F) 

The first sum is finite, and representatives can be chosen to be Weyl elements of H. Thus, 
if we define 

I w = / (p v {g) Y f T (wjug,s)i/;u (u)dudg 

C(F)\C(A) Uo(F)\Uo(A) je(w-lP(F) W r\P (F))\P {F) 

then / = J2 w ^ w wnere t ne sum is over the space of double cosets P(F)\H(F)/ Pq(F). 
Moreover, we can write the sum 

E = E E = E E 

j£(w--LP(F)wf)P (F))\P {F) ldQ w (F)\M {F) M G U${F)\U [F) ltQ w (F)\M (F) M 6 Uo, w {F) 
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Here Q w = Mo H w~ 1 Pw, which is a parabolic subgroup of Mo, Uq = Uo H w~ 1 Pw, and 
^e>,«j = ^e> H w~ 1 U~w, where ?7~ is the unipotent radical of the parabolic subgroup which 
is opposite to P. 

The next step is to consider the space Q w (F)\Mo(F)/C(F). We have 

E = E E 

1&Q W (F)\M (F) uEQ w (F)\M a (F)/C(F) ^(u-->-Q w (F)uf]C(F))\C(F) 

Plugging all this inside I w , we then collapse summation with integration. Thus, if we denote 



(26) I w ,u= J J ^A9)fr(wuug,s)ilj Uo (u)dudg 

(u-^Q w (F)uf]C(F))\C(A) Ug»(F)\U (A) 

then I w = J2 U ^w,u, where the sum is over representatives v for the set Q w (F)\Mo(F) jC '(F) , 
and U$ v = U a n (wv)~ x Pwv. 

Denote L v = v~ x Q w v[\C and V = {w^U^iwu)^ 1 . For v G V we denote 4>v( v ) = 
ipu ({wi , )~ 1 vwi') . Factoring the measure in the above integral, I WiV is equal to 
(27) 

J J/J F T (m(v)m(wul(wi')^ 1 ); wuug, s) 

MA)\C(A) C7g(A)\C7 (A) L„(F)\L V (A) V(F)\V(A) 

ip v (v)(pn(lg)' l Puo( u ) \8p(wvl(wi')~ 1 )\ s dvdldudg 

To explain the notation, we recall that the natural definition for an element of Indp^j t5 s p is 
as a function from H(A) taking values in the space of r<S>Sp. Since the space of r®5p consists 
of automorphic forms: M(A) — > C, an element of Indp^j t5 s p thus becomes a function 
M(A) x H(A) — > C, which we write F T (m; h, s). This function satisfies 

F T (m\, m 2 /i, s) = r ® 8 s p(m,2)F T (mi] h, s) = F r (mim 2 ; h, s). 

for all h G H (A), mi,m2 G M(A) and s G C. In order to define Eisenstein series, one passes 
to an alternate realization of Indp^s t5 s p consisting of complex-valued functions, by defining 
f T (h, s) = F T (e; h, s). See page 60 of [G-PS-R] for a further discussion. Also, in (1271) we have 
denoted the natural projection from the parabolic subgroup P to its Levi M by m. 
To proceed with the unfolding process we consider the following integral 

(28) J J ip 7 ,(lg)(^ T (m(v)m(wvlv^ 1 w^ 1 ))'il)v(v)dvdl 

L V {F)\L V (A) V(F)\V(k) 

It follows from the discussion after integral (12TI) that, as far as the unfolding process goes, 
unfolding integral ff28l) is equivalent to unfolding integral fT27|) . We emphasize that integral 
(|28|) is not an inner integration to integral (!27j) . 

Thus, we have reduced the computation to a lower rank situation. 
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We expect that for most elements v, integral ff27j) will be zero. We are interested in those 
cases where the following holds. 

Definition 29. We will say that that the global integral (1251) is an open orbit type 

integral if there is a unique element Wq G P(F)\H (F) / Pq(F) and a unique element in the 
space vq G Q w (F)\M (F) /C(F) such that the integral fTSBl) satisfies the dimension identity 

(30) dim 7r + dim r = dim L uo + dim V 

If (1301) is satisfied by more than one element u, but we can show that I Wjll vanishes for all 
but one, then we shall say that the integral ( 1251) is weakly open orbit type. 

To explain this terminology, consider the action of P x UqC on H by (p, ug) ■ h = ph^g)^ 1 . 
Then the stabilizer of wv is isomorphic to L v Uq v ', and in particular is of dimension dimL;, + 
dim V. The identity (T2"2"|) implies that 

dim 7r + dim r = dim Uq + dim C + dim P — dim H. 

This is equal to dimL u + dimV^ if and only if the orbit of wv has the same dimension 
as H, in which case it is open. Observe that an open orbit for this action (i.e., an open 
P, £/e>C-double coset), will certainly be contained in the open P, P^-double coset. Let wo 
denote a representative for this orbit, chosen to be the shortest element of the Weyl group 
which is contained in the orbit. Then the mapping Q wo vC — > PwqvCUq is a well defined 
bijection from Q W0 \M/C to the space of P, UoC-doub\e cosets contained in the open P, Pq- 
double coset. Hence, there is an open P, UoC-double coset in H if and only if there is a 
Q wo , C-double coset in Mq. 

Now assume that wv G H(F). (Recall that we identify H with H(F), where F is a fixed 
algebraic closure of H, and likewise for other algebraic groups.) Then H(F)nPwvlIoC may 
not be a single P(F) x Uo{F)C(F)-oibit. In cases where it is not, the integral (1251) is not 
of open orbit type, but it may still be of weakly open orbit type. 

Notice that the dimension identity given in definition [29] is analogous to the dimension 
identity (T2"2"|) . but this time the representations under consideration are 7r and r (as opposed 
to 7r and Indp^j r £g> 5 S P ). Experience indicates the following: 

Conjecture 31. Suppose that integral (|25j) is an open orbit type integral. Then integral (l27j) 
is zero, except when w = w o and v = Moreover, the element wq can be chosen to be Weyl 
element which corresponds to the longest Weyl element in the space P(F)\H(F)/Po{F). 

At this point our ability to check whether Conjecture [31] holds, is done only by carrying out 
the unfolding process. In addition, unfolding is, in general, necessary to determine whether 
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or not an integral is weakly open orbit type, in the case when Q WQ \Mo/C possesses an open 
orbit, which is a union of two or more orbits for the action of Q m (F) x C(F). 

We mention that all of the above is similar for integrals of the form (|14|) . The theta function 
which appears in the integral will appear in the same way in the analogue of integral (128]) . 
and definition [29] and conjecture [31] are similar with the addition of the theta function. See 
section [5731 for some details. 

We make one more definition. 

Definition 32. Suppose that conjecture [31] holds. We will refer to the integral (1231) as a 
preWhittaker integral if, for any representation r such that 0(r) is given in the above 
tables, there is a process of further unfolding of integral (128]) which involves the Whittaker 
coefficient of the representation tt. We shall also refer to an integral as weakly preWhit- 
taker if we are able to unfold the integral, but only by placing additional conditions on r 
(e.g., requiring that r be an Eisenstein series) which do not follow from the conditions on 
0{t). 

To prove that a given global integral is weakly preWhittaker, we only need to find a 
representation r such that integral [28] will unfold to a Whittaker coefficient in tt. This is 
frequently quite easy. Proving that a global integral is preWhittaker is more challenging, 
and generally requires some knowledge about the class of all representations of the Levi 
M attached to a given orbit. It is not clear how to prove that an integral is not weakly 
preWhittaker, or preWhittaker. Even though it is usually easy to guess when an integral 
does not unfold to a Whittaker coefficient in tt, it is not clear how to prove it. 

To satisfy the dimension equations, we require that r be a representation which supports 
certain Fourier coefficients corresponding to specific unipotent orbits. This is a method 
of sorting automorphic representations into classes which is convenient for our purposes. 
Another method is to sort the representations into cuspidal representations, Eisenstein series, 
and residual representations, and then sort the Eisenstein series according to which parabolic 
subgroup they are induced from, etc. Note that knowing the Gelfand-Kirillov dimension of 
a representation r, or even the more refined invariant Oq{t), does not tell us where r falls 
in this other classification. For example, a representation which is generic could be cuspidal, 
or it could be an Eisenstein series induced from any parabolic subgroup. However, it may 
happen that for some representation r integral (T28]) can be further reduced by means of 
Fourier expansions to a Whittaker type integral. See the examples given in sections [7] and ED 

There is one exception to the general principle that Orbcir) does not determine the type 
of r. 

Lemma 33. If Oq{t) is the singleton set containing the zero orbit, then r is a character. 
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Proof. One may express G as a quotient of G\ x • • • x Gk x T for some simple simply connected 
groups G%, . . . ,Gk and torus T, so the general case reduces to the case when G is simple and 
simply connected. 

We show that if ip is an automorphic form which does not support any Fourier coefficient 
attached to a unipotent orbit then p is invariant by U a (A) for all a, and hence invariant by 
(G,G)(A), where (G,G) is the commutator subgroup. 

To prove f/ Q (A)-invariance for all a, it suffices to prove it for one element of each Weyl- 
orbit in $(G,T), i.e., for one root of each length. If O m i n is the minimal orbit then Uo mln is 
a Heisenberg group with center U a for some long root a^. The fact that Fourier coefficients 
attached to this orbit vanish implies that <p is invariant by the center. This by itself treats 
the case when G is simply laced. In the case when there are long and short roots, this proves 
that p is invariant by U a {A) for all long roots a. Further, in each case one my identify a 

(2) 

second orbit O such that V Q , is the product of U as for some unique short root S, and 
possibly one or more root subgroups corresponding to long roots. Further, in each case the 
character ip v p)(v) = ip(v as ) is in general position. Since p does not support any Fourier 
coefficients attached to O' either, it follows that p is invariant by U a (A) for short roots as 
well. □ 



5.2. A Special Case. In this subsection we consider a special case which is valid for any 
group H. We explain the details for the case of integral ffl3|) . but it is clearly the same for 
integrals of the type given by ( 1141) . Let O be such that Pq is maximal. Let we denote the 
longest element of the Weyl group. Then wiMqw^ 1 is a standard maximal Levi subgroup. 
Let Pq be the standard maximal parabolic subgroup of H such that M = wiMqw^ 1 . (Among 
exceptional groups, Pq is equal to Pq except in some cases in E 6 .) We consider the integral 
( fl3l) in the special case when the parabolic subgroup P = MU from which we form the 
Eisenstein series E T (h, s) is equal to Pq. Thus r is an automorphic representation of M (A) 
and we assume that identity ([22]) holds 

(34) dimC + d\mUQ = dimn + dim£V(-, s) 

Unfolding the Eisenstein series, we consider the space P(F)\H(F)/Pq(F). Let wo be longest 
Weyl element in this space. From the relation P = Pq, we deduce that Uo jWo = Uq and 
that Q wo = Mq. Also, Wq 1 MqWq f] Mq = Mq. Hence, the space Q W0 \M o /C contains one 
element, and we define v$ = e. From this we deduce that integral ( 12 8)) is given by 

J <p v (t)(p T (t)dt 

C(F)\C(A) 
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We claim that identity ( 130|) holds in this case. Indeed, in this case, we have L Wo = C 
and V = {e}, where the last equality follows from the fact that Uq >Wo — Uq. Recall that 
dim E T (-,s) = dimr + dimU. We have dim [7 = dimUo- Plugging this into equation (|34p 
we obtain 

dim C + dim Uq = dim 7r + dim r + dim Uq 
Hence equation ( )30|) holds. 

We remark that integrals of this type are almost never preWhittaker. 

6. Maximal parabolic subgroups of F4 

The group F 4 has four standard maximal parabolic subgroups, Pi,P 2 ,P 3 and P 4 . (See 
section |2J) 



Levi 


Isomorphic to... 


M 1 


GSp 6 








M 2 


{(5-1,5-2) e GL 2 x GL 3 


det g 1 


det g 2 = 


1} 


M 3 


{(91,92) e GL 3 x GL 2 


det^i 


det g\ = 


1} 


M 4 


GSpin 7 









Before working an example it will be convenient to pin down a specific identification each 
standard maximal Levi subgroups with the matrix group listed above. 

We realize the group GSp 6 as the group generated by Sp 6 = {m 6 GL e : t mJ 6 m = J 6 } 
and the one dimensional torus diag(a, a, a, 1, 1, 1). Here 



-1 



V-i 

We fix specific isomorphisms by mapping 

/l r \ 
1 



Mi : 



x aA {r) ^ 



1 -r 
1 / 



/I 



1 r 
1 



1 — r 
1 



1/ 



/I 



1 r 
1 



1/ 
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(u 



^ y i{h)al{t 2 )a^{t2)ai{t A ) (->■ 



id ^3 



to t<} 



t 2 1 t 3 ti 



t 3 X t\ti 



M 2 



x ai (r) H> 



x Q3 (r) i-> 





M, 







ri r ^ 




x Q1 (r) 












V 11 













x a2 (r) ^ 




1 r 













«1 (*l)a2(*2)Q!3 {h)o:l{U) ^ 




t 1 X t 2 



^2 ^3/ 



^4^3 



The group M 4 is isomorphic to GSpin 7 , as mentioned above. This group has an embedding 
into S0 8 . It also has a projection to S0 7 which restricts to an isomorphism on unipotent 
elements. We define S0 8 relative to the matrix J 8 which has ones on the diagonal which 
goes from top right to bottom left and zeros everywhere else. We define to be the matrix 



with a 1 at the i, j and zeros elsewhere, and e'^ 
M 4 to GSpin 7 C SOg as follows: 



'9-j,9-i 



We fix an isomorphism from 



Ma 



(r) H- I* 



re' 12 + re' 



35' 



x, 



a 2 



V) ^ I 8 + re' 23 , 



x. 



(13 



V) ^ I s + re' 



34; 



a i (*i)«2 fa)^ (t 3 )a^ (U) i-> diag(tit 4 \ t x 1 * 2 , t 2 4i*3, ^ t 3 ^1*4, *2*i % **4, ^1*2 **4, *1 ^D- 
When working with unipotent subgroups of M 4 , we often find it more convenient to identify 
them with their images in S0 7 under a fixed projection. We define S0 7 relative to the 
matrix J 7 which has ones on the diagonal which goes from top right to bottom left and zeros 
everywhere else. We fix the projection M 4 — > S0 7 in such a fashion that 



(r 2 ) h-> J 7 + r 2 e' 23 , 



03 



(r 3 ) ^ J 7 



r 3 e 34 - r 3 e 45 - — e 35 , 



where now e^ - = — eg-^g-j. This determines the image of x a (r) for any positive root a, 
and we require that the projection of x^ a {r-) is t x a {r-) for each root a. This then determines 

30 



a unique map M4 — > SO7 which, as we say, is not an isomorphism but gives an isomorphism 
when restricted to any unipotent subgroup. 



7. An Example 

Let H = F4, and consider the unipotent orbit O = A2. The weighted Dynkin diagram for 
this orbit is 

ai (12 01.3 0:4 

==>== 2. 

From the table at the end of section HJ it follows that C is the exceptional group G 2 . We have 
Po = Mq -Uq, with Mq — GSpin 7 . Since this orbit is labeled with zeros and twos only, the 
integral to consider is integral ([TBI . Write Lq* for the rational representation of Mo which is 
dual to the rational representation of Mo on Lq given by conjugation. The F-points of this 
representation may be identified with the space of characters of Uo(F)\Uo{&)- The action 
of Mo on Lq* may be identified with the representation of GSpin 7 in which Spin 7 acts by 
the spin representation and scalars act by scalar multiplication. There is a nondegenerate 
quadratic form on L '* which is preserved by the action of Spin 7 . An element of L^*, (or its 
dual, the space of characters) is in general position if its image under this quadratic form is 
nonzero. Further, the elements of L '*(F) in general position are a single Mo(F)-orbit. See 
[S-KJ, pp. 114-116, and [I], §4 especially proposition 4. 

We choose the character ip Uo as follows. For u G Uo write u = Xnii(r 1 )x i2i(^2)^ / as in 
subsubsection [2707T1 Then ( x iiii( r i) x oi2i( r 2)w') = V ; ( r i + r 2)- It is not hard to check 
that this character corresponds to a point in general position. The embedding of G 2 into 
GSpin 7 as the stabilizer of ipu a is a pretty standard embedding. A maximal unipotent of 
this stabilizer is generated by all elements of the form 

^1000 (^ooioM, ZoiooM, 3:1100 0)^0110 (-?")> (r EG ) 

Xiiio(r)x 0120 (-r), x 1120 (r), x 122 o(r), 

For 1 < % < 4, we let w[i] be the reflection in the Weyl group attached to the simple 
root ccj. We use the same notation for the standard representative for this reflection, which 
is x ai {l)x- ai {— l)x ai {l). Finally, w[ii,i 2 , . . . ,ik] denotes the product of the corresponding 
simple reflections in the Weyl group, or of their representatives in H(F). Note that the Weyl 
group of G 2 may be identified with the subgroup of that of F 4 which is generated by the 
simple reflection w[2] and the product w[l,3]. 
Thus, integral ( fTBl) is given by 



1= j j tp n (g)E T (ug,s)i) Uo (u)dudg. 

G 2 {F)\G 2 (A) Uo(F)\UoW 
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The table at the end of section [4] indicates that each of the four parabolic subgroups of F4 
is an option for the parabolic subgroup P used to form the Eisenstein series. So we have 
four cases to check. In each case we carry out the unfolding process using the notations of 
section [51 

7.1. P = Pi. First we determine the space P(F)\H (F) / Pq{F) . A set of representatives is 
e, w[l, 2, 3, 4] and wo = w[l, 2, 3, 2, 1, 4, 3, 2, 3, 4]. Hence we obtain that Q WQ is the maximal 
parabolic subgroup of GSpin 7 whose Levi part is isomorphic to GLi x GSpin 5 . The group 
Uq,w 1S generated by elements x a (r), where a is in the set 



{(0001); (0011); (0111); (0121); (0122); (1122); (1222); (1232); (1242); (1342)}. 



Next we consider the space Q WQ (F)\ GSpin 7 (F)/G 2 (-^)- Consider the action of GSpin 7 (F) 
on the projective space consisting of all one-dimensional subspaces of its standard repre- 
sentation. Then Q Wo (F) is the stabilizer of the line consisting of all highest-weight vectors. 
Hence, the space Q Wo (F)\ GSpin 7 (F) may be identified with the orbit of this line. It is not 
hard to see that this orbit consists of all nonzero vectors "of length 0," i.e., on which the 
Spin 7 -invariant quadratic form vanishes. Now, Q WQ (F)\ GSpin 7 (F)/G 2 (-F) may be identified 
with the set of (^(-FO-orbits in this GSpin 7 (F)-orbit. But again one easily checks that G2(F) 
acts transitively on the nonzero vectors of length zero in the standard representation. Thus 
Q W0 (F)\ GSpin 7 (F) / G2(F) has only one element and we may take uq = e. 

Let R denote the parabolic subgroup of G2 which preserves the line of highest weight 
vectors in the action considered above (i.e., in the standard representation of G^)- Thus, R is 
the standard maximal parabolic subgroup of G2 which contains the SL 2 which is generated by 
£/±(oiqo)- Then in this case L uo = R. To determine the group V we first observe that U^ is the 
group generated by all U Q such that a is in the set {(1111); (1121); (1221); (1231); (2342)}. 
When conjugating by wq we obtain the group V which is the group generated by all U a such 
that a is in the set {(0001); (0011); (0111); (0121); (0122)}. Particular attention must be 
paid to Umi because it is not in the kernel of ipu- We have 

(35) w x lin (r)wo 1 = xoooi(^)- 

We identify the Levi part of P with the group GSp 6 as in section [HI and compute that 
m{v) is embedded inside GSp$ as the group Z which consists of all matrices in GSp$ of the 
form 



(36) 




32 



To describe the projection of R in GSp$ we first notice that wox±oiqow 1 = x±oioo- Also, 
w 1010 y (t)w = 1121 v (t). Here 1010 v (t) := a%(t)a%(t), and 1121 v is defined similarly. Hence 
the Levi part of R, which is isomorphic to the group GL 2 is embedded in GSp Q as 

\9\h 



9 



9 e GL 2 



As for the unipotent radical of R, which we denote by Ur, let 

(37) u = x W oo(ri)xo OW (ri)x 1W o(r2)xono(-r2)xnw(r 3 ,)x 01 2o(-r 3 )xn2o(r i )xi22o(r 5 ) 
Then conjugating by Wo, we obtain 

w uWq 1 = x 11 22(r 1 )xoow(ri)x 1 222(r2)x olw (-r 2 )xi232(r 3 )xo 1 2o(-r 3 )x 1 242(r4)x 134 2(r 5 ). 



Projecting into M, we get a; oio(^i)^oiio( 
6x6 matrices, we get 



m{wouw 



"^2)^0120 (~ r 3)- Using our identification of M with 



/I 



n r 2 

1 

1 



-r 3 

-n 
1 



V 

Combining all of the above, integral (1251) in this case is 



1 / 



/ / / 



ip v {ut)ip T {zm{wQUWQ 1 )t)^z{z)dzdudt 

GL 2 (F)\GL 2 (A) U R (F)\U R (A) Z(F)\Z{k) 

Here ipz is defined as follows. For z as described in (1361) . we set ipz(z) = i[)(xi } i). (It follows 
from (1351) that ^(2) = tpu ( w o lzw o)-) 

We count dimensions. We have L uo = R and hence dimL^ = 9. Also, dimU@ = 
dimV = 5. Hence dimL^ + dimU = 14. Since 7r is a generic cuspidal representation of 
G2, its dimension is 6, and as follows from the above table, we assumed that r is attached 
to the unipotent orbit (42) of GSp$ whose dimension when divided by two is 8. Hence 
dim it + dim r = 6 + 8 = 14. Hence (13171) holds. 

We state some conclusions. First, it is immediate from the discussion above that integral 
( 128]) is of open orbit type. Second, it is not hard to check that the contribution to the integral 
I from e and w[l, 2, 3, 4] is zero, so conjecture I3T1 holds in this case. 

By choosing r to be a certain Eisenstein series we can further unfold the last integral and 
obtain an integral with a Whittaker coefficient of tt. This shows that the integral is a weakly 
preWhittaker integral. See definition 5. Since we are not aware of any way to unfold the 
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above integral, to obtain a Whittaker coefficient of ir, and this for any representation r, we 
conjecture that the integral is not preWhittaker. 



7.2. P = P 2 . In this case the space P(F)\H(F)/Po(F) consists of 5 elements. We can 
choose as representatives the Weyl elements 



e; M2,3,4]; w[2, 1, 3, 2, 3, 4]; w[2, 3, 4, 2, 1, 3, 2, 3, 4]; w = w[2, 3, 1, 2, 3, 4, 3, 2, 1, 3, 2, 3, 4]. 



Hence, if we consider wq, then Q wo is the maximal parabolic subgroup of GSpin 7 whose 
Levi part M WQ is GL 2 x GSpin 3 . Also Uq° is generated by the elements x a (r) such that 
a G {(1221); (1231)}. The group Uo,w is the product of the groups U a for the other 13 
roots of T in Uq, which are 

0001, 0011, 0111, 0121, 0122, 1111, 1121, 1122, 1222, 1232, 1242, 1342, 2342 

Next we consider the space Q Wo (F)\ GSpin 7 (F) / G 2 (F) . This space can be identified with the 
set of Q Wo (F)-orbits in the GSpin 7 (F)-orbit of the character ipu - Recall that tpu * s identified 
with a point xq in the representation of GSpin 7 in which Spin 7 acts by the spin representation 
and scalars act by scalar multiplication. This is an eight dimensional representation of 
GSpin 7 . This representation has a Spin 7 - invariant bilinear form (, ). Assume that ( , ) is 
normalized so that (xq,Xq) = 1. Then the GSpin 7 orbit is Xu '■= {x : ( square}. 
For a suitably chosen basis, x = *(0, 0, 0, 1, 1, 0, 0, 0) and the image of Q m consists of 8 x 8 
matrices of the form 



Here and throughout, t A denotes the transpose of the matrix A over the diagonal which 
runs from top right to lower left. Thus t (°J) = We also introduce the notation 

g* := tp -1 -. Thus ( a c h d )* = a J_ bc {^ c d) ■ From this description it's easy to see that Xu{F) 
is a union of two Q Wo (F)-orbits: one consisting of those elements of the form t (x,y,z) with 
x, z E F 2 , y G -F 4 , such that 2^0, and the other of those such that z = 0. Clearly, the first 
of these two orbits is open. One may check that z/ = w[2, 1] is an element of it. In this case 
Lvo — z/ o" 1< 5«'o z/ o Pi ^2 = R° which is defined as follows. Take the subgroup R defined as in 
the case P = P\. Then R° C R consists of Mr = GL 2 and all elements u G Ur as in (l37j) 





where x' 



is determined by x, g x and g 2 , and g x ® g 2 is given by 
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with r\ = r 2 = 0. Thus dimLj, = 7. Thus, in this case, integral (125]) is given by 

</'7r(^lllo('"l)^012o(-^l)^112o(?'2)xi22o('™3)^X 



GL 2 {F)\GL 2 (A) (F\A)5 

( 38 ) /1 \ 

VnWVia I 1 2/1 J ip(yi)dyidrjdt 

Here we wrote r = 7~i x r 2 where ri is an automorphic representation of GL 2 (A), and r 2 is an 
automorphic representation of GL 3 (A). (Cf. section 0) Also, one can check that all other 
elements give zero contribution to /. Hence conjecture |3T] holds in this case. 

We count dimensions. We have dimL^ + dim = 7 + 2 = 9. According to the table the 
basic equation implies dimr = 3, and then formula (I3"0"j) holds as well. 

There are two possibilities for the pair (dimr!, dim r 2 ), namely (1,2) and (0,3). We now 
treat each of these two possibilities. 

7.2.1. dimri = l,dimr 2 = 2. If dimri = 1 and dimr 2 = 2, then T\ is generic and r 2 is 
attached to the orbit (21) of GL 3 . We will show that integral (1381) can be further expanded and 
unfolded to the Whittaker coefficient of 7r with the only assumption on r that 0(t 2 ) = (21). 
This will prove that integral fT2"8l) is preWhittaker in this case. Indeed, we have 
(39) 

T ri + 2 y 2 

Vt 2 I 1 Vi I ip(yi)dyidy 2 = \ ] I (p T2 I 1 i/i I ip(yi + az)dzdy 1 dy 2 



i n y 2 \ . h 

1 yi ^{yi)dyidy 2 = / ^ 



1 



If a 6 F*, then we obtain the Whittaker coefficient of r 2 as an inner integration. This is 
zero from the assumption that 0(t 2 ) = (21). Hence we are left with the contribution a = 0. 
Thus, changing variables, integral ( J38l) is equal to 
(40) 

J J V 5 7r(^1110(n)a;0120(-^l)^1120(^2)2;i220(^3)^)x 

GL 2 (F)\GL 2 (A) (F\A) 6 

A z y 2 \ 
VnWVTa I 1 2/i 1 ip(yi)dzdyidrjdt 

Next we expand ^ along the group Ur/Ur. We recall that Ur is the unipotent radical of the 
maximal parabolic subgroup of G 2 which preserves a line, and U R is the unipotent radical of 
the group R° defined above. The group GL 2 (F), as embedded in R° acts on this expansion 
with two orbits. The trivial orbit contributes zero by cuspidality. Thus, the above integral 
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is equal to 

(41) J J p n (ut)ij UR (u)duip Tl (t) J if 

B (F)\GL 2 (A)U R (F)\U R (A) (F\A)3 



1 z y 2 

1 Vi I ip(yi)dzdyidt 



Here ipu R is defined as follows. If we write u as a product given in (l3"7|) . then ipu R (u) = ip(ri). 
Also, in integral (j4"Tl) the group i?o is the subgroup of GL2 consisting of all matrices of the 



Id T \ 

form I J where a E F* and r E F. This group has the factorization B = TN where 

T is a torus and N is unipotent. In terms of the group _F 4 , the group N is identified with 
U i 00 . Factoring the measure in integral (|4"T|) we obtain 



(42) T(F)N(A)(F)\GL 2 (A) 



J J J ip^{ux mm {r)t)i) UR {u)ip Tl {x mm {r)t)drdu 



Ur(F)\U r (A) f\a 

1 z y 2 s 
rr, J 1 Vl I ' )</■:<///; 



(it x 



(F\A)3 

Next we plug the Fourier expansion of (p n along t/ i o into the expression in brackets. It is 
equal to 

^2 J J <P*( ux oioo(r + z)t)ipu R (u)ip(az)(p T1 (x i 00 (r)t)dzdrdu 

QeF (F\A) 2 U R (F)\U R (A) 

It follows from the cuspidality of 7r that the contribution from a = is zero. Plugging the 
summation in integral f)42p . collapsing summation and integration, we obtain the integral 



/ 



1 z y 2 

^s-M)W^ Tl {t)ip T2 I 1 Vl I (.'(//, )r/:r///,r// 



N(A)(F)\GL 2 (A) (F\A)3 

Here, for a vector ip a in a generic representation a we denote by its Whittaker coefficient, 
with the precise generic character of the maximal unipotent subgroup to be deduced from 
context. Thus integral fl28l) is preWhittaker in this case. 

7.2.2. dimri = 0,dimT2 = 3. (Equivalently, T\ is a character, and t 2 is generic.) In this 
case, we begin in the same fashion, by plugging in the expansion The group GL 2 (once 
identified with the Levi of the parabolic subgroup R C C) acts on the characters in this 
expansion with two orbits. That is, GL 2 normalizes Umo and if we define ip a {xmo{r)) = 
il>(ar), then %l) a (gxmo{r)g~ l ) = ip a det g (r). It follows that in this case I Wo ,v can be written as 
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Cn> + / io,n>> where 

C ,u = J J V ? 7r(^ino(n)^oi2o(-n)a;ii2o(^2)a;i22o(^3)fi')V ; (-n) drx 



GL 2 (F)V(A)\G 2 (A) (F\A) 3 

'1 z y 2 y 



(F\A) 




1 j/i I I (j. .s ■ I ( ■(/;-, ) //;,///.: f//y 




4 ,^o = y y V ? 7r(a;iiio(^i)a;oi2o(-n)a:ii2o(^2)a:i22o(^3)5')V ; (-n) drx 

SL 2 (F)V(A)\G 2 (A) (F\A) 3 

1 2/i | | 9, s | i/j(z + yi)dzdyidg. 

(F\A)3 \ \ \ 

Here V = ^1110^0120^1120^1220- 

Lemma 43. The integral J° „ o is zero for all cusp forms ty?^ and all / r e Indp^j r5p, where 
s G C and r is attached to the orbit (1 2 |3) of GL 2 x GL 3 . 

Proof. The proof relies on the fact that ip n is cuspidal, while f T is invariant by C/iooo- First 
expand 

^ ; = y 

V(F)\V(A) 

along 6 r iooo^ r iioo(-^ 1 )\^ r iooo^ r noo(A). The constant term in this Fourier expansion is a constant 
term of <p n , and hence zero. The remaining terms are permuted transitively by GL 2 (embed- 
ded as a subgroup of R). As a representative we choose Xi oo( r iooo) a; iioo( r iioo) ^ V , ( 7 "iooo)) 
and the stabilizer in GL 2 is the product of C/0100 and a one-dimensional torus. Thus 

Ti(F)[/ iooV(A)\G 2 (A) (F\A)3 V V V 1 / / / 

where V = Ui 000 U noo V, and ?/>y (v') = tp(v' 100Q ). Factor the integration over U 0100 (F)\U 0100 (A) . 
Since w is ■ 0100 = 1000, it follows that h >->■ f T (w u h, s) is invariant by f/ 0100 (A). And so 
the function ip n h-> ZjJ „ factors through the constant term attached to the standard max- 
imal parabolic subgroup of C whose unipotent radical contains f/oioo- This completes the 
proof. □ 

7.3. P = P3. In this case the space P(F)\H(F)/Pq(F) consists of 7 elements. We can 
choose as representatives the Weyl elements 

e; w[3,4]; ^[3,2,3,4]; w[3, 2, 1, 3, 2, 3, 4]; w[3, 2, 1, 4, 3, 2, 3, 4]; 
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w[3, 2, 3, 4, 3, 2, 1, 3, 2, 3, 4];w = w[3, 2, 1, 3, 2, 3, 4, 3, 2, 1, 3, 2, 3, 4]. 

The group Q wo is the maximal parabolic subgroup of GSpin 7 whose Levi part is GL 3 x GLi. 
A similar argument shows that the space Q Wo (F)\ GSpin 7 (F)/G2(-F) is finite and we may 
choose vq = w[3,2, 1]. Conjugating by wqI'o we obtain that L UQ = SL 3 embedded in G 2 as 
the group generated by the groups U a attached to the long roots. Hence, integral (12"8"]1 is 
given by 

^{t)i4> Tl {t)dt j f T2 y ^ i[)(x)dx. 

SL 3 {F)\SL 3 (A) F\A 

Here r = T\ x t 2 where T\ is an automorphic representation of GL 3 and r 2 of GL 2 . As in the 
two previous cases conjecture [31] holds in this case, and this is an open orbit type of integral. 



7.4. P = P 4 . This case is an instance of the special case discussed in subsection 15. 2[ In this 
case the space P(F)\H(F) / Pq(F) consists of the following 5 elements, 

e; w[% M4,3,2,3,4]; w[4, 3, 2, 1, 3, 2, 3, 4]; w = w[4, 3, 2, 1, 3, 2, 3, 4, 3, 2, 1, 3, 2, 3, 4] 

Thus Q Wo = GSpin 7 , and hence Q WQ (F)\ GSpm 7 (F) / G2{F) consists of one element. Thus 
uo = e, and integral ff28l) is 

ipir{t)ip T {t)dt 



G 2 {F)\G 2 (A) 

As before conjecture EH holds in this case, and this is an open orbit type of integral. 

8. Dealing with theta functions 

8.1. A lemma on Heisenberg groups. We often use the fact that some unipotent sub- 
group of F4 is isomorphic to a Heisenberg group. The point of this section is to record 
a simple lemma regarding what choices one has to make in order to pin down a specific 
isomorphism. 

By a Heisenberg group, we mean a two-step nilpotent, unipotent linear algebraic group U 
such that the center Z and the quotient U/Z are vector groups, and the skew-symmetric form 
U/Z x U/Z — > Z defined by the commutator is nondegenerate. The standard Heisenberg 
group "H2n+i in 2n + 1 variables is defined by equipping G" x G™ x G a with the operation 



(zi|yi|zi) • (x 2 \y 2 \z 2 ) = \xi + x 2 



yi +V2 



(For ( G 7^2n+i) we think of x and y as row vectors.) We normally write an element 

of "H2n+i as (x\y\z), with xjgGJ and z 6 G Q . We shall also use the notation (x|7/|z)2„,+i 
when it is desirable to reflect the size of the group in the notation. 



:i<S 



Lemma 44. If U is any Heisenberg group, then U is isomorphic to the standard Heisenberg 
group of the same dimension, and an F-isomorphism is uniquely determined by an ordered 
collection (xi, . . . , x n , yi, . . . , y n ) of injective F-morphisms G a — > U satisfying 

(xtir),!^)) = lu Mr),yj(s)) = l v Vi,j s.t. i + j^n+1, 

(yi( r ),yj(s)) = l u Vi,j, (xi(r),y n+1 -i(s)) = (xj(r), y n+w (s)) 7^ ly Vi,j, 

for all r, s G G ft . (Here, ( , ) denotes the commutator and Ijj denotes the identity element 
of the group U.) 

The proof is straightforward, and it is omitted. We note that an isomorphism U — > %2n+i 
also determines an isomorphism of the group of all automorphisms of U which fix Z pointwise 
with Sp2n- Suppose that a Fourier coefficient is defined as in section [3], by exploiting the 
Heisenberg group structure of U to define theta functions. Then the group C acts on U by 
automorphisms which fix Z pointwise, and hence a choice of isomorphism U — > "H2n+i also 
determines a homomorphism C — > Sp2 n - One may make this description more explicit in 
terms of the semidirect product T-L2n+i *> Spin- given an isomorphism I : U — > %2n+i> there 
is a unique homomorphism p : C — > Sp 2n such that l(gug~ l ) = p(g)l(u)p(g~ 1 ) for all g G C 
and u G U. 

Write $(U/Z, T) for the set of roots of T in the quotient U/Z, i.e., for $([/) \ $(Z). The 
simplest way to choose a family of morphisms in lemma HH is to 

• pair up the roots of T in U/Z: for each a in <&(U/Z,T) there is a unique "partner," 
such that the sum the unique root of T in Z. 

• order the roots in such a way that for each i, the root at position In + 1 — % is the 
partner (as above) of the root at position i, 

• for i — 1 to n, let Xi = Xp il where Pi denotes the root at position i in the ordering 
just fixed, 

• for i = 1 to n, let y { (r) = xp 2n+1 ^(r/N((3i, (3 2n +i-i))- 

Here, N(fti, fan+i-i) denotes the structure constant, defined as in |G-S] . This will be our 
"standard" method of determining an isomorphism U — > 1-i2n+i based on listing the roots of 
T in U/Z in a specific order. Note that the roots of T in U/Z come equipped with a partial 
ordering associated to the base of simple roots for $(G,T). That is, a < (3 if [3 — a is a 
positive root. We shall normally order the elements of U/Z in a fashion which is compatible 
with this partial ordering. 

As an example, consider the unipotent orbit Ai in the exceptional group F4. It's diagram 

is 

Qi C*2 &3 CK4 

1 ==>== 0. 
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The corresponding parabolic subgroup P is P\. The unipotent radical U of this parabolic 
subgroup is isomorphic to "His- Indeed, it has a one dimensional center, which is the root 
subgroup £72342- The other 14 roots of T in U are 

1000, 1100, 1110, 1120, 1111, 1220, 1121, 1221, 1122, 1231, 1222, 1232, 1242, 1342. 

One easily checks that for each root a which appears on this list, the "partner" 2342 — a is 
also present. For example, the partner of 1120 is 1222. Also, the roots are ordered in such 
a way that each root and its partner are in symmetric positions. For example, 1120 is the 
fourth root, and 1222 is fourth-last. In order to fix an isomorphism I : U — >■ H15, we require 
that I (£2342 (z)) = (0|0| 2 ) and 

^(^1000(^1)^1100(^2)^1110(^3)^1120(^4)^1111(^5)^1220(^6)^1121(^7)) = (ri,r 2 ,r 3 ,r 4 ,r 5 ,r 6 ,r 7 \0\0). 
The relevant structure constants from |G-S] are 
iV(1000, 1342) = -1, JV(1100, 1242) = 1, iV(1110, 1232) = -2, JV(1120, 1222) = 1, 

JV(1111,1231) = 2, iV(1220, 1122) = -1, JV(1121, 1221) = -2. 
This means that 

(ziooq(?"), £1342(5)) = ^2342(-rs), (xn2i(r),xi22i(s)) = x 23 42 (-2rs) , etc. 
It follows that the preimage of (0|yi, y 2 , ys, 2/4, 2/5, Vq, J/7I0) under / must be 

(-|) «u»(-»0*». (f ) x 1222 (, 4 )x 1232 (-|) WaOW-i*). 

Also in this case there are six other orderings of the roots which are compatible with the 
partial ordering inherited from our choice of a base of simple roots: 

1000, 1100, 1110, 1120, 1220, 1111, 1121, ... 

1000, 1100, 1110, 1120, 1111, 1121, 1122, ... 

1000, 1100, 1110, 1120, 1111, 1121, 1220, ... 

1000, 1100, 1110, 1111, 1120, 1220, 1121 ... 

1000, 1100, 1110, 1111, 1120, 1121, 1122 ... 

1000, 1100, 1110, 1111, 1120, 1121, 1220 ... 

Where ". . . " indicates that the remaining seven roots are ordered by symmetry. 

In practice, it makes sense to vary the choice of ordering (i.e., the choice of isomorphism 
Uq —> W-zn+i) based on the situation. 

Returning to the general case, suppose that if Zi,Z 2 : U — > 7i2n+i are two isomorphisms, 
defined over F, such that the restrictions to the center of U agree. Define pi,p 2 '■ C — > 
Sp2n as before, by requiring that Pi(g)h(u)pi(g^ 1 ) = l^gug^ 1 ) for i = 1,2. Then there is 
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an element a G Sp2 n (F) such that l2{u)p2{g) = ali(u)pi(g)a^ 1 . Hence Ot(h(u)pi(g)) = 
9u i} (a)<j>(h( u ) Pzid)) ^ or an § S(A n ). Thus, one may vary the choice of isomorphism U — > 
%2n+i fairly freely. 

We shall refer to an algebraic group U as a generalized Heisenberg group if 

(1) it is two step nilpotent, with center Z equal to its commutator subgroup (U, U), 

(2) both Z and U/Z are "vector groups," 

(3) there exists a linear form £ : Z — > G a such that the skew symmetric bilinear form 
U/Z —> & a induced by composing £ with the commutator U — > Z is nondegenerate. 

Note that the existence of £ as in the last condition clearly requires dim U/Z to be even, and 
that any choice of £ induces a projection from U onto U / ker £, which is a Heisenberg group. 

Lemma 45. Let O be an orbit and define Vq as in section [31 Suppose that £ : vffi /Vq^ — > 
G a is in general position. Then (x,y) i— > £(xyx~ 1 y~ 1 ) is a nondegenerate skew-symmetric 
form on Vq^ /Vq\ 

Proof. To explain this phenomenon, one needs to review a bit more of the theory under- 
pinning the construction of Fourier coefficients given in section [3j Let O be a unipotent 
conjugacy class in H. Then, O can be identified via the exponential map with an orbit for 
the adjoint action of H on its Lie algebra 1). Such an orbit can be identified, via the Jacobson- 
Morozov theorem, with an orbit of Lie algebra homomorphisms — > f). Let L : 8l% — > f) 
be a homomorphism in the orbit attached to O, with the property that hi := L (J \ ) lies 
in the Lie algebra t of our fixed maximal torus T. Then the weight of a simple root a« in 
the weighted Dynkin diagram of O is simply the eigenvalue for hi on the one- dimensional 

(2) 

subalgebra t) ai . Furthermore, L can be chosen so that £(expX) = (3 (X, fi) , for all X e X> ' 

(2) 

(the Lie algebra of Vq ), where fi '■= L (° (j) • Write f)j for the z-eigenspace of h L in I). By 
viewing f) as a direct sum of L^sh)- modules, we see that X i— > [/l,X] is an isomorphism 
f)i — > f)_i. Now, given I e |i there exists Y G such that (3(X,Y) ^ 0. But then for 
X' G fji with Y = [f L ,X'\, we have (3(X, [f L ,X'\) = /3([X, X'], f L ) ^ 0. Nondegeneracy of 
the form (x, y) i-> £(xyx~ 1 y~ 1 ) follows easily. □ 

The procedure outlined above for fixing a specific isomorphism from a Heisenberg group 
to the standard Heisenberg group of the appropriate dimension may be used on generalized 
Heisenberg groups to extend £ to a projection I : U —> 'H.2n+\ for suitable n, if the function 
£ : Z — > G a is supported on a single root subgroup U a . 

As an example consider the unipotent subgroup U = U4 of F±. It is two step nilpotent and 
15-dimensional. The center, Z, is seven- dimensional, and we have 

$(Z, T) = {0122, 1122, 1222, 1232, 1242, 1342, 2342}. 
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The group U/Z is eight-dimensional, and we have 

$(U/Z, T) = {0001, 0011, 0111, 1111, 0121, 1121, 1221, 1231} 

Observe that for each a G $([/ /Z,T), the root 1232 — a is also in <&(U/Z, T). Hence we can 
define a surjective homomorphism I :U —± Tig such that 

I (Xo 1 22(r 1 )xn22(r 2 )x 1 222(r 3 )x 1 232(r4 : )Xi242(r 5 )Xi 3 42(r 6 )x2342(r 7 )) = (0|0|r 4 ). 

As in the case of an ordinary Heisenberg group, there are many such isomorphisms, and to 
pin down a specific one, we may use a choice of ordering on the roots and the homomorphisms 
x a , as well as the associated structure constants. For example we may order <&(U/Z,T) as 

0001, 0011, 0111, 1111, 0121, 1121, 1221, 1231. 

Note that for each root a, the "partner" root 1232 — a is in a symmetric position. The 
relevant structure constants are 

iV(0001, 1231) = -1, JV(0011, 1221) = 1, JV(0111, 1121) = 1, JV(1111, 0121) = -1. 

So, we may define I : U —> Tig such that 

U^oooi (ri)x oii(r 2 )xoiii (r 3 )x m i (r 4 )) = (ri, r 2 , r 3 , r 4 |0|0), 

l{xQi2i{r 1 )x 1 i2i{r 2 )x 1 22i{rz)x 1 2zi{r A )) = (0| - n, r 2 , r 3 , -r 4 |0). 
In this case, there is only one other possible ordering, which is 

0001,0011,0111,0121,1111,1121,1221, 1231. 

Note that the above construction does not work if 1232 is replaced by one of the other roots 
in $(Z,T): for each other root (3 G $(Z,T) there exists a G $(U/Z,T) such that (3 - a is 
not a root. On the other hand, one can check that 

H X 0122(Z 1 )X 11 22(Z2)X 1 222(Z 3 )X 1 232(Z4)X 1 242(Z5)X1342(Z6)X2342(Z 7 )) = (0\0\z 1 + Z 7 ) 

^(^0001(^1)^0011(^2)^1111(^3)^1121(^4)) = (ri,r 2 ,r 3 ,r 4 |0|0), 
^1221(51)^1231(52)^0111(53)^0121(54)) = (0| - 2si,2s 2 , -2s 3 , -2s 4 |0), 
gives a well defined homomorphism I : U — > Hg. (Note that 

^0122(^1)^1122(^2)^1222(^3)^1232(^4)^1242(^5)^1342(^6)^2342(^7) ^ Z4 

and 

^0122(^1)^1122(^2)^1222(^3)^1232(^4)^1242(^5)^1342(^6)^2342(^7) ^ z \ + z l 
are in general position for the action of M 4 = GSpin 7 on the space of linear maps Z — > G a , 
but 

^0122(^1)^1122(^2)^1222(^3)^1232(^4)^1242(^5)^1342(^6)^2342(^7) l— > z i 
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is not for i ^ 4.) 



8.2. Explicit formulae for the Weil representation. In this short subsection, we record 
for convenience some explicit formulae for the action of the Weil representation of T^n+iCA) x 
Sp 2n (A) which we shall use repeatedly when dealing with cases involving theta functions. A 
reference is jP]. We write Mat° xn for the space ofnxn matrices X such that t X = X. 
For x, y G A n (realized as row vectors), z G A, g G GL n (A),e = ±1, and X G Mat° xn (A) : 

(46) c^(O|O|z)0 = ^(z)<P c^(x|O|O)0(O = 0(£ + x) c^((%|O)0(O = V(l/ *O0(O 

(47) uv(P J )£ U(a;) = £7det s 0(^) fW) = V> ( k^W), 



9 , 

where 7 a is given in terms of the function u defined on p. 377 of [?] by the formula 7, 
w(l)w(-l/a). 

Lemma 48. If k < n are integers, embed Sp 2 k ^ 5 , P2n and %2k+i ^ %2n+i by 

(z|y|*)2k+i | -> (0,ar|y,0|«)a,n.i. 




This induces an embedding t : %2k+\ X Sp 2k {.A) "%2n+i X Sp2 n (A) sucn that ^ 6 = 
Here o;^ and denote the Weil representations of T^n+i x Sp 2n (A) and %2k+i x Sp 2k (A), 
respectively. If is an element of the Schwartz space S(A n ), u G H 2n+ i(A), and g G Sp 2n (A), 
define 

(49) ^|>S0 : = ^ ^(^0(0,0- 

Then 

(i) : 

J 0j((O|O,j/|O)u5)dy = O^ug) (Vn G H 2n+1 (A),g G ^ 2n (A)). 

(F\A)"" fc 

(ii) : For u G %2fc+i(A),g G Sp 2k (A) 1 we have 

^ u (t(ttflf)) = % s *(u3f) ) 

where 6^ 2fc is the theta function on T-L2k+i x Sp 2k (A) defined using the function 
0i(x) := 0(0, x), which lies in S(A h ). Here x G A k and denotes the zero element of 
A n ~ k . 
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(iii): The function 0^, satisfies 



(so) e 



'4> 

k 



9l \ ( In-k X Y 

hk X' | ug 

In—k 




7det 9l #J fc M, 



for all gi E GL n - k (A),u E H 2 n+u9 £ Sp 2n (A), and for all X E Mat( n _ fe ) x(2 fc)(A), 
X' E Mat(2fe)x(n-Jfc)(A)) Y £ Mat( n _jfc) X ( n _fe)(A), such that the matrix above is in S^n- 

Proof. This follows from the formulas for the Weil representation given above. □ 

8.3. Unfolding in the presence of theta functions. In this subsection, we give a dis- 
cussion similar to section 15.11 for an integral of the type defined in equation (fT4l) . As in sub- 
section EIH we express the integral (1T11) as a double sum indexed first by w E P\H/P , and 
then by v E Q W \M/C, with Q w = MHw^Pw. For v E M we define L v = Hr\{wv)- 1 P{wu) 
and V = Q C\ wUw^ 1 . Then in this case 



(g)fr(wvug, s)9%(l(u)p(g))ip Uo (l'(u)) dudg 

L V (F)\C(A) U$»(F)\UoW 

= J <Pn{g) J J f T (wuu 1 ug,s)9 , /l ) (l(uiu)p(g))i) Uo (l'(u 1 u)) duidudg. 

L V (F)\C(A) U^(A)\UoW U$" (F)\U$" (A) 

Here I and I' are the projections of Uq/Vq"' onto its Heisenberg and abelian factors as in 
section HI The definition of "open orbit type" for an integral of type (JHJ), of course reflects 
the dimension of the theta representation. 

Definition 51. An integral of the type (j!4p is said to be of open orbit type if there is a 
unique Wq E P\H/Pq and a unique u E Q Wo \M/C such that 

dim L UQ + dim Uq oV ° = dim it + dim r + dim 6^. 

It is weakly open orbit type if the number uq satisfying this identity is greater than one, but 
the integral I Wo ,u vanishes for all v§ but one. It is preWhittaker if 

(1) it is weakly open orbit type, 

(2) I w v vanishes for all pairs (w, v) ^ (w , z/ ), 

(3) I Wo ,v is equal to a Whittaker integral, with no conditions on r beyond those listed 
in the table at the end of section HI 

It is weakly preWhittaker if this the first two conditions hold, and I Wo ,u is equal to a 
Whittaker integral, under some additional condition on r (e.g., that it is an Eisenstein 
series) . 
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In this case, we shall refer to the integral I Wo ,u as the "contribution from the open orbit," 
and, introduce a corresponding "inner period," analogous to ([25]) . and given by 

(52) J (pnv{h) J (p T (m(wi i uh(wiy)~ 1 ))8^(l(u)p(h))ijju (l' (u)) du dh. 

The integral I w v can be simplified in some cases using lemma HHJ 

It should be noted that for integrals of this type, it is sometimes necessary to take 7r to be 
a genuine representation on a covering group of (7(A). In this case, uniqueness of local Whit- 
taker models, and as a consequence Whittaker integrals need not be Eulerian. Thus, the ques- 
tion of whether a given integral is pre Whittaker may not be directly relevant to the question 
of whether it is Eulerian. Nevertheless, particularly in view of [BBCFHJ,[BBF1J,[BBFHJ,[], 
it seems that preWhittaker integrals which involve a cusp form on a covering group are 
certainly not without interest. 

9. An example with a theta function 

Take O = A x . The corresponding weighted Dynkin diagram is 

ai a% «3 «4 

1 ==>== 0. 

Hence Pq = Pi- The unipotent radical of P\ is a Heisenberg group in 15 variables. The 
center is t/ 2 342- The Levi, Mq is isomorphic to GSp 6 . 

Define a character of Tlffi by ijju (u) = V ; ( M 2342)- The stabilizer C of this character in Mq is 
the derived group of Mo, isomorphic to Sp$. We identify it with Sp$ using the isomorphism 
Mo — » GSpQ given in section El 

Write 0^ for the theta representation of "His (A) x Sp u (A), and let I : Uo — > "His be an 
isomorphism such that I (0:2342 (z)) = (0\0\z). It will be convenient to choose different iso- 
morphisms / for different cases below. The isomorphism I also determines a homomorphism 
C —> Spu. Since the action of C on U0IU2Z12 is equivalent to the third fundamental rep- 
resentation of Spe, we denote this homomorphism by zu 3 . See |S-K] . pp. 107-108 [I], §5 for 
some discussion of this representation. Note that on the level of matrices, W3 depends on I, 
which will vary from case to case. 

The image of Spq(A) in Spu (A) does not split under the covering map Sp 14 (A) — > Spu(A). 
Thus an element of 0y is a genuine function defined on the double cover of (7(A) = Spe(A), 
which we denote (7(A). In addition let Fa(A) denote the metaplectic double cover of F 4 (A) 
defined using the Matsumoto cocycle. Then U(A) x (7(A) <— y F 4 (A). 

The options in this case were discussed in detail in section HI In brief, since each vector 8^ 

in the space of O7 restricts to a genuine function on Sp 6 (A), our global integral should include 
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either an Eisenstein series which is genuine and a cusp form which is not, or an Eisenstein 
series which is not genuine and a cusp form which is. If P — Pi or P 3 then dimr = 0, which 
means that r is a character, and this is incompatible with E T being genuine. Therefore, 
in these cases it must be the cusp form which is genuine. If P = P\ or P4, then as far as 
we know both options are available. However, our arguments will be the same regardless 
of which function is genuine. For concreteness, we assume that the cusp form is genuine in 
these cases as well. Thus, our integral is 



where <p m is a cusp form in the space of some genuine irreducible cuspidal automorphic 
representation of Sp 6 (A), and E T is an Eisenstein series defined on the group P^A), and &t 
is a theta series from the representation Qj of %i 5 (A) x Sp 14 (A). 

Referring to the unfolding process sketched above, notice that C contains every unipotent 
element of Mq. It follows easily that for all standard parabolic subgroups P we have 



In particular, the integral is of open orbit type for all P, and one can take v to be the 
identity in all cases. Since Uq is the identity, it follows that L VQ = Q WQ D C. This is a 
parabolic subgroup of C and it's Levi part is equal to the intersection of C with the Levi of 



9.1. P = Pi. This case is an example of the phenomenon discussed in subsection 15.21 In 
this case, w = w[l, 2, 3, 2, 1, 4, 3, 2, 1, 3, 2, 4, 3, 2, 1], while Q wo = Mq = M 1 . As already 
mentioned, the integral (TT4"]) is of open orbit type for this Fourier coefficient, regardless of 
the parabolic subgroup used to form the Eisenstein series. Further, vq = identity, L UQ = 
M = Mi, and V is trivial. Hence, I WQ)V0 is given by 




P\H/CU = P\G/P = W{M,T)\W/W(M ,T). 



Q 




C(F)\C(A) UoW 



while the inner period (|52|) is given by 




C(F)\C(A) 



Lemma 53. Conjecture I3T1 is satisfied in this case. 



Proof. The space P\H/ CUo = P\H / Po is represented by 



e,w[l], Ml, 2, 3, 2, 



1], ^[1,2,3,2,4,3,2, 



1], and wq. 
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Let w denote one of these representatives, and let 

I w = J (p„(g) J f T (wug,s)9^(l(u)w 3 (g))dudg. 

Q0(F)\S P6 (A) U$(F)\U (A) 

Here Q° w = Sp e fl w~ l Pw. We must show that /„, = for all w ^ w . 

If w = e or w[l], then w ■ 2342 G &(U, T) which means that f T (wug,s) is invariant by 
^2342 (A) on the left. This clearly forces the integral I w to be zero. 

If w = w[l, 2, 3, 2, 1], then Q° is the maximal parabolic subgroup of Sp& with Levi isomor- 
phic to Spi x GL\. Fix an isomorphism of XJq with "H 15 by ordering the roots as 

1000, 1100, 1110, 1120, 1220; 1111, 1121| 1221, 1231; 1122, 1222, 1232, 1242, 1342. 

Here, extra spacing has been used to reflect the way that Lq decomposes as a direct sum of 
three invariant subspaces under the action of the standard Levi subgroup of Q w . Also, the 
middle of the list is marked with a vertical bar. This will be standard practice. 

Note that the three Mo-invariant subspaces actually form a flag which is preserved by the 
action of the unipotent radical of Q w . Thus, if R is the maximal standard parabolic of Spu 
which has Levi isomorphic to GL 5 x Sp^, and if w 3 is defined by ordering the roots as above, 
then w 3 will map the Levi of Q w into the Levi of R and the unipotent radical of Q w into the 
unipotent radical of R. 

Then wa G &(U, T) for all of the last five roots. Therefore, by lemma I4H| 

(p v (g) J f T {wug,s)9^ 2 {l{u)w z {g))dudg, 

Q° W (F)\S P6 (A) Ug(F)Vi(A)\Uo(A) 

where V\ is the five dimensional abelian unipotent subgroup of U corresponding to the last 
five roots listed above. 

Since both h h-» f T (wh) and h h> 9^Aw 3 (K)) are invariant by the unipotent radical of Q° w , 
the integral I w vanishes by the cuspidality of tp n . 

Finally, suppose w = [1,2,3,2,4,3,2,1]. Then Q° is the maximal standard parabolic 
subgroup of Spe whose Levi subgroup is isomorphic to GL 3 . In order to analyze /„, in this 
case, we shall use a different isomorphism U — > "His, obtained by ordering the roots as 
follows: 

(54) 1000; 1100,1110,1111,1120,1121,1122| 1220,1221,1222,1231,1232,1242; 1342. 
Thus 

^1000(71)^1100(72), £1122(77)) = (n, r 2 , r 3 , r 4 , r 5 , r 6 , r 7 |0|0) l{x 2 3^{z)) = (0|0|z), 

I (^1220 (2/l)^1221 (y2)£l222 (y3)£l231 (2/4)3;i232 (2/5)^1242 (^6)^1342 (2/7) ) 

= (%i, -2y 2 , 2/3, 2j/ 4 , -2y B , y s , — J/t|0). 
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Since w ■ 1342 = 1342 6 $(U,T), lemma HH] implies that 



I w = J <p w (g) j f T (wug,s)6J la (l(u)zu 3 (g))dudg. 

Q%(F)\S P(i (A) U%(F)Uvu2W\UoW 

Next we unfold the partial theta function 

The standard Levi subgroup of Q° w , isomorphic to GL%, acts on F 6 (which may be iden- 
tified with the abelian subgroup of Uq corresponding to the six roots 1100, . . . 1122) by a 
representation which is equivalent to the symmetric square representation of GL3. Choose a 
collection S of orbit representatives, and for each representative £ let denote the stabilizer 
in GL 3 . Let N denote the unipotent radical of Q° w . Then we have 

/« = £ j <pM J [<^i>( l ( u )^3(g))<P}(0,^)f T (wug,s) dudg. 

^ eS ( (F)N(F)\Sp 6 (A) U£(F)U 1342 (A)\Uo(A) 

Let N'(F) denote the six- dimensional abelian subgroup of C/o(A) corresponding to the roots 
1220, 1221, 1222, 1231, 1232, 1242. Then wN'tv' 1 = N. One the other hand, wNw' 1 lies in 
U. Factoring the integration on N and N', we find that 

I w = ^2 J ffi'^ig) J [u^(l(u)w 3 (g))(f))(0,^)fr N ' qps \wug,s)dudg, 

^ S ( (F)N(A)\S P6 (A) U% (F)N' (A)U 134 2 (A)\U (A) 

where 

Zpf^\g)= J ipn(ng)ipt(ri) dn, \wug, s) = J f T (nwug, s)^(n) dn, 

N(F)\N(A) N (F)\N (A) 

and and iff't are two characters of iV which depend on £ 6 F G . We need to calculate this 
dependence precisely 
First, take 

n'{y) = Xi220(l/i)a;i22i(y2)a;i222(y3)a;i23i(y4)a;i232(z/5)a;i242(y6) 

Then 

K n '(y)) = (%i> ~ 2 V2, Vs, %4, -2y B , ye, — 2/r|0), 

so 

[fcty(n'(y)/i)0i](O,f) = 4>(y t = ^{yxb - ^2^ + 2/3^5 + 2j/ 4 £ 4 - 2j/ 6 £ 3 + j/e6)> 
for £ = (£ 2 , • • • , £7) € -F 6 - On the other hand 

wn'(y)w~ 1 = xo W o(y 1 )xoiio(y2)xoi2o(y3)xoiii(yi)x 01 2i(y5)xo 1 22(y6), 
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which is identified with 

h 





I 2/4 


-2/5 


2/6 


e Sp 6 , 


Y=\-y 2 


2/3 


-2/5 




\ 2/1 


-2/2 


2/4 



'IT 3 ^) = r(Tr(E •!•)). whore E = | 4, 6; 



Thus 

^4 & 6 
^3 £5 ^6 
^2 £3 £4, 

In order to describe fy, one needs to compute the restriction of zu 3 to N. Let 

^ij &ij ^15— j',15— ij ^ij "I - ^15— j, 15— i" 

Here e^- denotes the 14 x 14 matrix with a 1 at i,j and O's everywhere else. Thus Spu 
contains I14 + rej- if 1 < i, j < 7, Iu + re"j iil<i<7<j< 15 — i, and J14 + re^ if 
i + j — 15. (Here, J14 is the 14 x 14 identity matrix.) We have 
(55) 

W^onoM) = J i4 - re' 13 - r 2 e" 8 - 2re^ + 2re'l 9 , zj 3 (x 0100 (r)) = I u + re' 12 + re^ - 2e 69 , 
^3(^0111 (r)) = Ji4 - re' 14 - r 2 e" 10 + 2re£ 9 - 2re4 10 a^omM) = iu + r e' 15 + re 2 ' 8 - 2re 4) u 
ro 3 (x i2i(r)) = J14 + re' 16 - r 2 e" 13 - 2re 29 + 2^3^ ^(zomO")) = Iu + re' 17 + re 2 10 - 2re 3i i 2 

It follows from (147jl . (|55]l . that 



J 



9 



0(0, 6, •••,&) 



3 



= ^(-2(^5 - + 2(66 - - - 66)*i 

+ 2(66 - k&Ki - (£ 4 2 - 66)^2,2 - (£ 6 2 - 

X (C7g (#)) 0(0, &,•••,&)■ 

Thus ^ ^ 3 = '0(— Tr(E ad X)) where E is as above, and S ad is the matrix whose i, j 

entry is the determinant of the i,j minor of E. 

Now, if S is trivial or of rank one, then S ad is trivial, and hence cpl^'^^ vanishes identically. 
On the other hand, if S is of rank three, then f T ' e is a Fourier coefficient of f T , which is 
attached to the orbit (2 3 ). According to the table, r is attached to (2 2 1 2 ), so fi vanishes 
in this case. 

This leaves the set of symmetric matrices of rank two. This set is a single orbit under the 
action of GL3, and we may choose the matrix S such that ^6 = 1 and the rest are zero as a 
representative. 
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We claim that is invariant by the group f/ooio^ooii f° r this choice of £. To see this, 

consider the Fourier expansion 

a,b£F 



fi N "^°' b \g,s) := J /f^ ) (x 00 io(ri)a;ooii(r 2 )^ S )^(ar 1 + 6r 2 )rfr. 

(F\A) 2 

If a and b are both nonzero, then f? N '^' a ' f, - ) is a Fourier coefficient attached to the orbit (42) 
of SpQ. Such a coefficient vanishes identically on the space of r. If one of a, b is zero and the 
other is nonzero, then after conjugating by w[3] if necessary, we may assume that a = and 
b is nonzero. Then one may rewrite f^ N '^ s,a ' b " > as an iterated integral with the inner integral 
being 



fi v ^\g, s) := J f T (vg, s)Mv) dv, V=l ('* X X'Y. X = ^ 1 c Sp 6 , 

V(F)\V(A) IV J 




J 2 X' = ^(6X1,1 + X 2 ,2) 



Now, let V be the group defined in the same manner as V but with X being arbitrary. 
Then every extension of ipy to a character of V is in general position. Integration over V 
against a character in general position is a Fourier coefficient attached to the orbit (3 2 ). Such 
a Fourier coefficient vanishes identically on the space of r. From this we deduce that f( v rtv) 
vanishes identically, and thence so does /| W ^°' b \ 

This leaves only the constant term in the expansion of fr along t/ooio^oon? which 

(N,M ) 

proves that f T ' e is invariant by this group. This permits us to express I w as an iterated 
integral with the inner integral being the constant term of (p n along the maximal parabolic 
subgroup of Sp 6 with Levi isomorphic to GL 2 x SL 2 . Thus I w vanishes. □ 
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9.2. P = P2. The basic data in this case is as follows: 

Wo = w[2, 3, 2, 1,4, 3, 2, 1,3, 2, 4, 3, 2,1] 
M m =GL 3 x GLi (An$(M„ ,T) = {a 3 ,a 4 }), 
1/0 = identity, 

•^1^0 c n Q W Q 

= the maximal parabolic subgroup of Sp6 with Levi = GL 3 



^0 = ^ 



1342 



V = r„ 1(l(1 ^ ( 1 *l,/ 3 ^ (-U > GL> x Ct'L.s) 



dimr = 0. 



Here, M W(i is the Levi factor of Q Wo . 

Fix an isomorphism / : U —¥ H15 as described in section [8j based on ordering the roots as 
in (154]) . We consider the restriction of the corresponding embedding w 3 : Sp e — > Spu to the 
parabolic subgroup L uo . First the Levi factor of L uo is isomorphic to GL 3 , and we may write 

1*3(9) = diag(det0,sym 2 (0),sym 2 (0)*,det0 _1 ), 

where the 6x6 matrix sym 2 (g) denotes the matrix of g acting on the symmetric square 
representation for a suitable choice of basis. 

Since r is a character in this case, f T will be left-invariant by V. Hence, the integral I WQjVo 
for this case is 

J J ^n(g)fr(w ug,s) J 9^(l(x 1342 (r)u)w 3 (g))drdgdu. 

(7i342(A)\C/ (A) (P 2 nC)(F)\C(A) F\A 

By lemma HHJ part (i) 



( pA9)fr(w ug,s)eJ lfi (l(u)ui 3 (g)) dudg, 

(P 2 nC)(F)\C(A) U 1S42 (A)\Uq (A) 

where 6^ is defined as in Lemma 1451 The corresponding inner period (152]) is given by 



GL 3 (F) \GL 3 (A) Matg x 3 (F) \ Mat g x 3 (A) 



ijV' 

I 
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h X\ g 

h { tg- 1 



dX dg, 



(Keep in mind that r is a character.) Identify Mat 3x3 with the unipotent radical of C H P 2 
via the mapping 14 ( 73 f ) and the isomorphism Spe — > C. Also, identify GL 3 with the 
Levi of C fl P 2 via the mapping g — >■ ( 9 tfl -i ) and the isomorphism Sp 6 — > C. 
Observe that for g 1 G GL 3 (F),g G S , p 14 (A), if £' := (£ 2 , • • • ,£7) • sym 2 (gi), then 



h X 



0(0,0 = Uty ( W S 
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/ 3 x 



9 



7 3 g x X t gi 
h 



9i 



t9i 



h 
1 J (J 



0(0, 6, • ••,&) 
0(0, 6, •••,6)- 



Hence, if we write S * g\ for the matrix analogous to S corresponding to the vector 
(62, • • • , £7) • sym 2 (5(i), then we have 

MTr((Z*g 1 ) ad X)) =^Tr(E ad ( gi X tgi ))) = ^{{ t g l ^ d g l )X)) = ^(Tr((^S ^) ad X)). 

It follows that the polynomial identity S * g± — g{E t gi holds for all g± G GL 3 (F),2 G 

Mat° x3 (F). 

Now unfold the theta function, identify (£2, • • • ,£7) G P 6 with S G Mat 3x3 (P), and split 
the sum over S up into orbits for the action of GL 3 (F). It is clear that rank(S) is an invariant 
for this action. Define 

I*o„>*= J I J ^(( /3 ^)^(H ad X)rfX 

Stab H (F)Matg x3 (A)\5p6(A) E/1342 (A)\t/i (A) Mat§ x3 (F)\ Matg x3 (A) 

f T (w ug, s)u^(l(u)w 3 (g))<l>(0, £2, • • • , £7) du dg, 
where Stab- denotes the stabilizer of S in GL 3 . 

Proposition 56. The integral I WOjVOj s vanishes unless S is of rank three. 

Proof. If S is trivial or of rank one, then S ad is trivial and the assertion is obvious. Suppose 
then that S is of rank two. We may assume that £ 2 = £3 = £4 = ^6 = 0, since each 
GP 3 (P)-orbit contains elements with this property. The stabilizer Stabs then contains the 
two-dimensional unipotent subgroup i/ooio^ooii which corresponds to 




cGL 3 . 



Since 



^3(zooio(?"i)zooii(r 2 )) = d+ne2^+r2e 2A +r 1 e 25 +r 1 r2e' 26 +r 2 e' 27 +2r 1 e 35 +r2e' 36 +r 1 e 46 +2r 2 e[ 7 , 

it follows that u>^(l(u)w 3 (g))(f)(0, 0, 0, 0, £ 5 , 0, £7) is invariant by C/ooio^oon- Since r is a charac- 
ter, it follows that f T is also invariant by i/ooio^Wi- Further, the conditions we have placed on 
£ imply that Tr(S ad X) = -£ 5 &X3,i- Factoring the integration over lWo^oon(P)Wooio^ooii(A), 
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we obtain the constant term of <p w along the parabolic subgroup of Sp$ with Levi GL 2 x SL 2 
as in inner integral. Hence, I m , VOt s vanishes in the rank two case as well. □ 

In view of proposition ESI we may write 

Iw ,v 

with the sum being over representatives for the distinct orbits of nondegenerate symmetric 
matrices. This is similar to an integral where P x UqC has an open orbit X in H such that 
X{F) is a union of infinitely many P(F) x Uo{F)C (F)-orbits. Further, there is no apparent 
reason why any of the terms in the above sum should vanish. 

For the sake of completeness, we verify conjecture EH in this case as well. 

Lemma 57. Conjecture I3T1 holds in this case. 

Proof. There are seven elements of P\H/CUo, represented by the Weyl elements 
e, w[2,l], M2,3,2,l], ^[2,3,2,4,3,2,1], w[2, 1, 3, 2, 4, 3, 2, 1], 

w[2, 1, 3, 2, 1, 3, 2, 4, 3, 2, 1], w[2, 3, 2, 1, 4, 3, 2, 1, 3, 2, 4, 3, 2, 1]. 

Define I w as in lemma [531 The first three Weyl elements listed above map 2342 into the 
unipotent radical of U. It easily follows that I w is zero for all choices of data in these cases. 

To study the other cases, let Q° = C fl Q w , (a parabolic subgroup of Sp e ). For w = 
[2, 3, 2, 4, 3, 2, 1], the group Q° has Levi isomorphic to GL 3 . Identify Uq with H15 using (154"|) . 
Then the last seven roots are all mapped into $>(U,T). As in lemma |4"8| this kills every 
nonzero term in the sum over £ G F 7 which defines 9t , leaving a function which is invariant 
by the full maximal unipotent subgroup of Spe- Factoring the integration over the unipotent 
radical of Q w , we obtain the constant term of £v along this unipotent radical. This proves 
that I w = in this case. 

If w = w[2, 1, 3, 2, 4, 3, 2, 1], then the situation is similar. Indeed, w ■ a is positive for 
a = 1342, 1242, 1232, 1222, 1122, 1221, 1222. From the table, we know that r is a character 
in this case, and hence, f T is again invariant by the seven dimensional unipotent group 
corresponding to these seven roots. 

Finally, we consider w = w[2, 1, 3, 2, 1, 3, 2, 4, 3, 2, 1]. For this case, the standard Levi sub- 
group Q° w of L° w is isomorphic to GL 2 x SL 2 . Further, wot > for a = 1231, 1232, 1242, 1342. 
Let V\ denote the product of the groups U a for these four roots. Define / and zu 3 by ordering 
the roots as follows: 

1000, 1100, 1110, 1111; 1120, 1121, 1122|1220, 1221, 1222; 1231, 1232, 1242, 1342 
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Then using lemma HHJ one obtains 



I w = j <p w (g) j 9^ 3 (l(u)vj 3 (g))f T (ug,s)dudg, 

Q0( F )\ Sp6 (A) V 1 (A)\U (A) 

further, g i-> 6^ 3 (zu 3 (g)) is invariant by the unipotent radical of Q° w , as is g i-)- f T (wg,s). 
Factoring the integration over this group, we obtain zero in this case as well. □ 

9.3. P = P3. The basic data for this case is 

wo =w[3, 2, 1,4, 3, 2, 1,3, 2, 4, 3, 2,1] 

GL 2 x GL 2 (A n $(Mu, ,T) = {a 2 , « 4 }), 
z/ = identity, 

= the maximal parabolic subgroup of Sp$ with Levi = GL 2 x SX2 
m{wQV L vo v Q l w l ) ^ { ((^ 1) '^2^ : 9i,92 e GX 2 ,det0i = det^ 2 j , (M m> GL 3 x GL 2 ) 

[/ = f/i242^1342 

V" = C^OlOO^llOO 

1 Xi 

(M ^GL 3 x GL 2 ) 





Lemma 58. Conjecture [31] holds in this case. 

Proof. Define I w as in lemma [531 The set P\H / CUq has five elements, wq and the following 
four others: 

e, io[3,2, 1], io[3,2,4,3,2,l], w[3, 2, 1, 3, 2, 4, 3, 2, 1]. 

Once again, by the table, we know that r is a character. The first three coset representatives 
above map 2342 to a positive root. It follows that I w vanishes in all these cases. 

Now assume w = w[3, 2, 1, 3, 2, 4, 3, 2, 1]. Then wa > for a = 1122, 1222, 1231, 1232, 1242, 1342. 
Order the roots of T in Uo/(Uq, Uq) so that these six roots come last. Then 



w 



V>M / fr(wug,s)el h (l(u)w 3 (g))dudg, 



Q°(F)\S P6 (A) U$(A)\U a {A) 

and both g 1— > f T (wg,s) and g h-> ^^(^(fiO) are invariant by the unipotent radical of 
which proves that /„, is zero, since (p n is cuspidal. □ 
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We now proceed to analyze I Wo . Using once again the fact that r is a character, we obtain 



I I 



fiMMwoug, s)6^(ug) dgdu. 

?7i24aE/i34a(A)\f/o(A) (P s nC)(F)\C(A) 
Define I : U — >• % 15 and w 3 : Sp 6 — > Spu using the ordering given in equation 
Identify with Sp 6 using the isomorphism fixed in section [61 Recall that L uo is a standard 
parabolic subgroup of Spe in this case. The Levi factor of L UQ is isomorphic to GL 2 x SX 2 and 
consists of all matrices of the form diag(#i, g 2 , t9\ 1 )- We identify this group with GL 2 x SL 2 
via the map (<7i, (72) l— diag(<7i, #2, tfl'r 1 )- For elements of this group ?u 3 is given by 

diag(#i,# 2 , ^r 1 ) H> diag(det c/i-^, 0i, p(gi,92), tQi 1 , det g^ l - t g 2 l ), (g t G GL 2 ,g 2 G SX 2 ) 

Here, p(gi,g 2 ) is a certain 6x6 matrix. 
Plug in 

^i b m = E kM^i (o,o,e 3 ,...,e7), 

and split £ up into £' = (£3, £4) G -F 2 and £" = (£5, £6, £7) G F 3 . For <? in the GL 2 factor of 
the Levi of L uo , we have 073(0) = diag(det g, det 5-, g, s 2 (g), t s 2 (g)~ 1 , t#~\ det#~\ det^ -1 ), 
where 



a b 
c d 



a 

. 2ac be + ad 2bd 

ad ~ bc \ c 2 cd d 2 



Thus GL 2 (F) acts on {£' = (£3, £4) : £3, £4 G F} with two orbits. The contribution from the 
trivial orbit is 



(59) 

C/1242C/1342 (A)\Ui (A) (P 3 nC) (F)\C(A) 

To see that this is equal to zero, one checks that 

f (h 



J ( P7r(g)fT(woug,s)6^ 3 (ug)dgdu. 




C Spe maps into < 



* 



* 
* 



* 
* 
* 



{ \ 



A 1 

* 
* 
* 
1 



> c Sp 



2 ) J 



ii- 



under the embedding w 3 : Sp§ — » Spu, defined using the ordering given in (J5"3j) . It follows 
from lemma|l8]that the function 8^,(ug) is invariant by this group, and, since r is a character, 
f T (w ug,s) is invariant as well. It follows that the mapping ip n \-t I mMh o, where I WO)Vo ,o is 
defined by equation (159]) . factors through the constant term of (p n , and hence vanishes. 

To describe the contribution from the open orbit, let So,i denote the stabilizer of £' = (0, 1) 
in GL 2 (still identified with a subgroup of L UQ as above). It consists of all matrices of the 
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form ( a * ) . Also, S'o i i\G r I/2 may be identified with P 2 \Ly , where 



P 2 ° := ( 



(a * * * * \ 
1 * * * 

1 * 



G S]?6 : a G GLi, ft, G SX 2 



It follows that 



£ £ ^(flO0 1 (o,o,r,n= E E ^(75001(0,0,0,1,0, 

?'eF2\{(0,0)} e'6-F3 7 eP 2 (F)\L„ (F)?"eF3 

for all # G Sp u (A) and 0i G S(A 7 ). Plugging in, 

'"•„./'„ - / / £ ^(^'^(sO^O, 0, 0, l,£")f T (w u'g, s)du dg. 



P°(F)\Sp 6 (A) JU> 



Here [/' = £/i 2 42 ^1342^0- 

Next, it follows from (jUD and (ESD that 



/ 



(60) uty 



U7 3 



V 



(\ q r\\ 

1 p q 

1 

1 

1 

\o ij J 



0! (o,o,o,i,e") = ^(-p)0i (0,0,0,1,0 



for all p, q, r G A, all f" G F 3 , and all 0i G S(A 7 ). 

We shall write Z for the three dimensional unipotent group considered in (1601) . because it 
is the center of the unipotent radical of L uo . Then h h-> f T (woh, s) is left-invariant by Z(A). 
It follows that I Wo ,u is equal to 



/ / 



(61) 



P°(F)Z(A)\S P6 (A) 



<Pn(zg)ipz(z) dz 



Z(F)\Z{k) 



U'(A) 



£ ^(/K)w3^))0(o, 0,0,1,0 



f T (w u'g, s) du'dg, 



where i[>z( z ) — ^{—p) in the coordinates of (1501) . 
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Now, the function J Lp n (zg)ijjz{z) dz is invariant by Y(F), where 

Z(F)\Z(A) 



(62) 



Y=ly 



/l a b q 

1 p 

10 

1 

1 

\0 



r \ 

Q 
b 

—a 


1 / 



e Sp 6 > 



Hence we can apply Fourier expansion on the group Y (F) Z (A)\Y (A) . The subgroup of P 2 ° 
consisting of matrices of the form diag(a, 1, h, 1, a -1 ) : a G GLi, h G SL 2 acts on the space 
of characters of Y (F) Z (A)\Y (A) with 2 orbits. We claim that the trivial orbit contributes 
zero. The term corresponding to the trivial character is 



(63) 



P 2 °(F)Z(A)\S , p 6 (A) JU'(A) 



{yg)^ Y (y) dy 

) (F)\Y(A) 

^2 ^v( / ( M ') ro 3(5 , ))0(O, 0, 0, 1, C)fAw u'g, s) du dg, 



where ip Y * s the trivial extension of ipz to a character of Y. In the coordinates of f )62|) . it is 
given by ip Y (y) = ijj(-p). 



Lemma 64. The integral (|63|) vanishes identically. 



Proof. In the proof we shall repeatedly use the formulae for given in f|4*6|) and fj47|) . 

Set ffi^\g) = / (p n (yg)ip Y (y) dy. Factor the integration over F(F)Z(A)\F(A), 

Y(F)\Y(A) 

which may be identified with L r oon^ r oiii(-P 1 )\^ r oon^oiii(A). Note that f T and (p w ' Y are 
both invariant by this group on the left. One computes 



^(tU3(xoiii(r 3 )xooii(r2)))0(O,O,0, 1,£ 5 ,£ 6 ,£ 7 ) = T/>(-2r 3 £ 5 )0(O, 0, 0, 1, £5,66, £7 + 2r 2 ), 
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z deducing that 



^2 J Ui,(l(u')vu 3 (xoni(r 3 )x 00 n(r 2 )g))(j)(0, 0,0, 1, £5, £ 6 , £ 7 ) 

(F\A) 2 ?" eF V'(A) 

f T (w u'x 0111 (r 3 )x oii (r 2 )g, s)^ { I^ y) (x m (r 3 )x 00 ii (r 2 )g) du' dr 

= iLi ^(w 3 (xoni(r 3 )xooii(r 2 ))l(u')w 3 (g))(j)(0, 0,0,1, 

(65) (f\A) 2 €" eF V'(A) 

f T (w x om (r 3 )a;ooii (r 2 )u'g, s)^'^ (x m (r 3 )x 00U {r 2 )g) du dr 
[ ^-2r 3 &H(*(«'W0))0(O, 0, 0, 1, £ 7 + 2r 2 ) 



(F\A)2 «" eF V'(A) 

/r (w x iii (r 3 )x on (r 2 K#, s)^'^ (xoin (r 3 )x 00 ii (r 2 )50 du' dr. 

As indicated in the table above, in this case r is a character. Since Wq ■ 0111 and Wq ■ 0011 
are both positive, it follows that ^(^0^0111(^)^0011 (r 2 )u'g, s) = f T (wou'g, s). Also it follows 
from the definition of Lp^^ Y ^ that ^^'^(xom^s^ooii^)'?) = ft^'^id)- Collapsing the 
summation on £7 with the integration on r 2 , and applying Fourier inversion in r 3 and £5, we 
find that fl65l) equals 

u^{l(u)wz{g))(j){0, 0, 0, 1, 0, f 6 , r 2 ) dr 2 f T (w u'g, sjcp-n ' Y (g) du' 

U'(A) A 

Next, u7 3 (x 00Q1 (r)) = I+re 34 +re5 6 +2reg 7 +r 2 e5 7 , hence w v ,(ccr 3 (a;oooi(^)))0(O, 0,0, 1,0,^ 6 , r 2 ) = 
(f)(0, 0, 0, 1, 0, f 6 , r 2 - 2r£ 6 ), and so 

^ u^{l(u')w 3 (g))(t)(0, 0, 0, 1, 0, f 6 , r 2 ) dr 2 f T (w u'g, s) du' 

17' (A) A ^ 6eF 

is invariant by C/oooi- Factoring the integration, one obtains an expression for (|63|) as an iter- 
ated integral, such that the inner integration is the constant term of (p^ along the unipotent 
radical of the parabolic subgroup of Spe whose Levi part is GL\ x Sp±. It then follows that 
the integral (E3D is zero. □ 



We continue to study the expansion of (15T1) along Y(F)\Y(A). Next we consider the 
contribution from the nontrivial orbit. Choose the character ipy(y) = ip(a — p) (in terms of 
the coordinates given in (|62l) ) as a representative for this orbit. Then the stabilizer inside 
GL\ x SL 2 as embedded in Sp 6 is the semidirect product of T\ = diag (a, 1, a, a -1 , 1, a -1 ), 
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and the unipotent group Iq + re^. Thus the integral (JBTJ) is equal to 



(66) 



/ 



<p*(y9)i>Y{y) dy 



(A)T 1 (F)C/Sg(F)\Sp 6 (A) JU'(A) Jy(F)\Y(A) 

^2 u}^(l(u')w 3 (g))<j)(0, 0, 0, 1, £")f T (w u'g, s) du dg. 



£"£F 3 



Here, t/^ax * s the standard maximal unipotent subgroup of Spe. Next we factor the integration 
over Uom(F)\Uom(A) . As a subgroup of Sp$ this group is 



/I 








m 







> 





1 






















1 








m 













1 








> 














1 







\o 














ij 





Note that 



¥n(yg)^Y(y) dy 



lY(F)\Y(A) 

is invariant by this group, as is f T . Using ( 1551) . one can check that 

w v ,(tz7 3 (x O iii(m)))0(O,O,0, 1,^5,^6,^7) = ^(-2m^)0(O,O,0, 1,^5,^6,^7)- 
Integration over m then picks of the term £5 = 0. Hence (166]) equals 



/ / 

Z 1 (A)T 1 (F)U^XUF)\SP6(A) U'W 



ffi^Hg) Yl ^(^K)^3(^?))0(o,o,o,l,o,r)^(w,s)^ , ^, 



©3 (^0010(C6)^001l(6/2)) 



1 > 



where /i 



L : 


— f^ooio^oon- 


We compute 




/I 








\ 




1 6 6/2 


a ^7/2 


e?/4 




1 


2£e 6/2 







1 


6 








1 









1 





V 






1 / 



Hence, by (H7|> . 

w^(c7 3 (flf))0(O, 0,0, l,0,f 6 ,£r) = X] ^( ro 3(7flO)0(£o), 

yeUooioUoouiF) 

where 6 — (0,0,0,1,0,0,0). Collapsing summation with integration, the above integral is 
equal to 



(67) 



Z 1 (A)T 1 (F)Vi(F)\Sp 6 (A) </£/' 



(pij^^uj^u'^^g^^^f^w^'g^s) du' dg, 
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where 



Vi 



(\ a b c d\ 

1 e f * 
1 g * * 



\ 



J 



(a subgroup of f/^al complementary to [/ooio^oon)- Here, and in what follows, we exploit the 
fact that an element u of t/^ax is determined by the entries Uy with l<i<j<7 — i. Now, 
it is easily checked that the function h i— > f T (woh) (h G F 4 (A)) is left Vi(A) invariant. (Here, 
we identify V\ with a subgroup of F4 using the identification of M4 C F± with GSp$ given in 
section [6j) Using ( H7|) and (1551) one can check that that the function h u^(h)4>(^,o) (h e 
Sp 6 (A)) is invariant by the groups [/0001(A), [/0100(A), and [/0110(A). It then follows that 



9^ 



U'(A) 



f T (w u'g, s)u^(l(u')w 3 (g))<j)(^ ) du' 



is also invariant by [/0001(A), [/0100(A), and [/0110(A). The product of these three groups may 

be identified with Z!(A)\Vi(A). Therefore, when we factor the integration over V\{F) Z 1 (A)\V L (A) , 

the integral (1671) is equal to 



^(1/2^)^2(1/2)^2 



'Ti(F)Vi(A)\Sp 6 (A) JY 2 (F)\Y 2 { 

where 



U'(A) 



^{u'^{g))^ Q )f T {w Q u'g) du' dg, 



/l a h b c d\ 
1 e / * 
1 m * * 



V 



v 2 



/l a h b c d\ 
1 e f * 
1 m * * 



J 



\ 



J 



Using the left invariance property of <£v under elements in Sp 6 (F), we deduce that 

vAyzQ)^ (2/2)^2 



~(Y 2 ,</>y 2 ), v 

<P* (9) ■-- 

'Y 2 (F)\Y 2 ( 

^{V2w[2>\g)^Y 2 (2/2)^2, 

'r 2 (F)\r 2 (A) 

where w[3] is our standard representative in Ng{T) for the simple reflection s a3 G W(G,T), 
and ipY 2 (y2) = i ) Y 2 {w[3]~ 1 y2iv[3]) . One checks that 



y 2 



(\ a h b c d\ 
1 e f * 
1 g * * 



V 



ip(a-g). 



J 



Finally, the function <p n ' Y2 is invariant by [/ooio(-^)- We plug in the Fourier expansion 
along this group. The term corresponding to the trivial character vanishes by cuspidality. 
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The remaining characters are permuted transitively by the action of a one dimensional torus 
which is conjugate under w[3] to the torus T\. Thus, in the end we obtain 



W^{w[3]g)<jj^{u'w 3 {g))^ )f T (w u , g)du , dg. 

'Vi(A)\5p 6 (A) JU'(A) 

We conclude that this case is preWhittaker. 

9.4. P = P4. The basic data in this case is 

w = w[4, 3, 2, 1,3, 2, 4, 3, 2,1] 

M Wo = GLi x GS Pi (An $(M W0 ,T) = {a 2 ,a 3 }), 

z/ = identity, 

L UQ c n Qwq 

= the maximal parabolic subgroup of Sp$ with Levi = GL\ x Sp^ 



Sp[ a2 ' a3} ■ ol&GL 



m{w Q L VQ w 1 ) = Sp{ a2 ' a3} ■ 2321 V (GL!) C M = GSpin 7 

U W ° = ^1122^1222^1232^1242^1342 
V = ^1000^1100^1110^1120^1220 

= the unipotent radical of the maximal parabolic subgroup of 
M = GSpin 7 , with Levi = GL X x GSpin 5 . 
Here 2321 v denotes the cocharacter of T given by 

2321 v (t) = a^(t 2 )a^(t 3 )a^(t 2 )a v 4 (t), t G G m . 
Lemma 68. Conjecture I3T1 holds in this case. 

Proof. The set P\H /CUq has only three elements, represented by the identity, wo, and 
w := [4,3,2,1]. The identity maps 2342 into U which proves that the term corresponding 
to this double coset is zero. Let Q° = Sp e fl w~ 1 Pw. It is the standard maximal parabolic 
subgroup of Sp 6 with Levi subgroup isomorphic to GL 2 x SL 2 . 

Fix the homomorphisms I : Uq —> H15 and zu 3 : Sp 6 — > Sp u by ordering the roots of 
Uo/{Uoi Uo) as follows 

1000,1100; 1110,1111; 1120, 1220, 1121|1221, 1122, 1222; 1231,1231; 1242,1342. 

The last four roots are sent to $([/, T) by w. Write V w for the corresponding unipotent 
subgroup. Then applying lemma 1451 we have 



I 



w 



&n(g) I f T (wug,s)e^ l3 (l(u)vj 3 (g))dudg. 



QS,(F)\Sp 6 (A) U$(F)V W (A)\U (A) 
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Further, both g (->■ f T (wg,s) and g h-> 0^{w^{g)) are invariant by the unipotent radical of 
Q%, so I w vanishes by cuspidality. □ 

To analyze I WQ)V0 , it is more convenient to define the isomorphism I : U — > H15, and the 
embedding zu 3 : Sp$ — > Spu by ordering of the roots of T in U / (U, U), thus: 

1000, 1100, 1110, 1120, 1220; 1111, 1121|1221, 1231; 1122, 1222, 1232, 1242, 1342. 

(Recall that changing the isomorphism I changes the bijection </>—>■ 9% between Schwartz 
functions and theta functions, but not the space of functions obtained.) It follows that 
(69) 

Z -1 (0|y|0) = X122I (-y) ^1231 (yj ^1122 (-2/3)^1222 (2/4)2:1232 (~y) £1242(2/6)^1342 ("2/7), 

for y = (yi, y 2 , . . . , yi). Having identified C with Sp$, the group C fl M 2 is identified 
with {diag(a, g, a -1 ) C Sp6 : ot G GL±,g G S 1 ^}. If g G Sp4, we may write ^(g) = 
diag(Ag(g), g', t(^o(g))), where A^g) denotes matrix for g acting on the five-dimensional 
fundamental representation of Sp^ with respect to a suitable basis, and g' = tgt^ 1 , for 
t = diag(-2,-2,l,l). 
We may then write 

J II ^^^J^ l ' 2/ l ) Mon73 ^)^ r ^^) WoMo5, '' s ) dydgdu , 

U Wo (A)\U(A) (CnP 2 )(F)\C(A) (F\A) 5 

where 

(70) v(-y 1 , -y 2 , -y 3) -2/4, -ys) = ^1000(^/1)^1100(2/2)^1110 (y) 2:1120 (-2/4)2:1220 (2/5) 

satisfies l(wQ 1 v(y)w ) = (0|0, 0, y\0) G "His, for y = . . . , j/ 5 ). The image of u(y) under the 
isomorphism M — > GSpin 7 fixed in section [6] and the natural homomorphism from GSpin 7 
onto 5*07 is 

/I 2/l 2/2 2/3/2 t/4 2/5 * \ 

1 -2/5 

1 -2/4 

1 -2/3/2 G S0 7 . 

1 -2/2 
1 -2/1 

V 1 / 

We now expand f T along the group V. The group S^" 2 ,Q3 ^(F) • 2321 v (GLi(F)) acts on 
the characters of V(F)\V(A) with infinitely many orbits. Indeed, the space of charac- 
ters of V(.F)\V(A) may be identified with the F points of the rational representation of 
Spf 2 ' a3 * (F) ■ 2321 v (GLi(F)) which is dual to the quotient of V by its commutator subgroup 
(V,V). We denote this space by V/{V,V)*. Then 2321 V (GL!) acts linearly on V/{V,V)*, 
and the action of Sp\ a2 ' a3 ^ on V/(V,V)* can be identified with the action of SO5 on its 
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standard representation (regarding Sp 4 as Spin 5 ). Identify V/(V,V)* with column vectors 
and let Q denote the nondegenerate Spin 5 -invariant quadratic form on this space. Then 
the orbits for Sp 4 • GLi are as follows: one orbit consists of the zero vector, and a second 
corresponds to all vectors i ^ such that Q(x) = 0. The vectors x with Q(x) 7^ split into 
infinitely many orbits parametrized by square classes in F x . That is, two vectors X\, X2 with 
Q(xi)Q(x 2 ) 7^ lie in the same orbit if and only if Q(xi) /Q(x 2 ) is a square. The zero vector 
of course corresponds to the trivial character. As representatives for the remaining orbits 
we take 



and 



WMy)) = "0(2/2 - 02/4)}, 



where a ranges over a set of representatives for the nonzero, nonsquare square classes in 
F. (If a is a square, then ipy, defined as above, is equivalent to ipy.) The stabilizer of ip v 
in Sp\ a2 ' a3} (F) ■ 2321 V (F X ) is SL% 3 (F) ■ 2421 v (F x )f/ 0100 lWo^oi2oOF), with the cocharacter 
2421 v being defined in the same way that 2321 v was defined above. The space SX^-F) • 
2A21 s/ {F x )U owo U olw Uoi2o{F)\Sp\ a2 ' a3} {F) ■ 2321 V (F X ) can be identified with Pj}(F)\(C H 
P 2 )(F), where 



(a * * 
a * 
h 

V 



* \ 

* 
* 



a G GLi, h G SL 2 



> C Sp e 



The stabilizer of ip v in Sp\ a2 ' a3 ^ (F) ■ 2321 V (F X ) is isomorphic to the semidirect product of 
SL 2 x SL 2 and a group of order 2 which acts by reversing the factors. Specifically, the 
stabilizer contains SL?! 2 , SL® 120 , and suitable (nonstandard) representatives for the simple 
reflection w[3] in W{G,T). 

If a is not square, then the stabilizer of ip v in Spf 2 ' a3} '(F) ■ 2321 V (F X ) also has two 
components, but the identity component is isomorphic not to SL 2 ,, but to Res F r/^/ F SL 2 . 
Here F(y/a) denotes the unique quadratic extension of F in which a is a square, and 
Res F (^y F denotes restriction of scalars. If we identify Mi with GSp$ as in section El 
then Sp\ a2 ' a3 \F) ■ 2321 V (F X ) corresponds to all matrices of the form 



a 



a G GLi, g G Sp4, 
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c«.; 









g 2 a gi g 4 a g 3 
95 96 97 9s 



while the identity component of the stabilizer of ip v consists of all matrices of the form 

9i9i + ag 2 gs ~ 9395 ~ ag^ge = 1, 

9i9s + 9297 - 939e ~ 9495 = 0. 

A representative for the second connected component of the stabilizer in this case is 

diag(-l, -1,1, 1,-1,-1), 

which acts on the identity component in a manner which corresponds to the mapping h h- > 
thT 1 , with being the action of the nontrivial element of Gal(F(y/a) / F) . 
From this discussion, it follows that 

i = i 00 + 1° + j 1 + v r 



aeF x/ F x,2 



where 

rOO 

U WQ (A)\U(A) (CnP 2 )(F)\C(A) (F\A) 5 



fr(h,s)= J f(v(y')h,s)dy', 

(F\A) 5 



r 



= y / / ^(^)<((0|0,t/|0) M o^3(^))/f' <) (^(l/)^o^,s) rfy^^o, 

l/ W0 (A)\C/(A) P 3 °(F)\C(A) (F\A) 5 



C/ TO0 (A)\C7 (A) 5L^2 ( F ) SL oi20 (F)(Cnf7 2 )(F)\C(A) 

P (V,^) 



(F\A) 



5 



^((0|0,y|0)^3(<?))/)™ (u(?/)«;o«o^,s) dydgdu , 



U WQ (A)\U(A) S a (F)(CnU 2 )(F)\C(A) 



(F\A) 



5 



Where S a is the stabilizer of ip v , described above, and 

f^y\h,s)= J f T (v(y')h,sW v (v(y))dy', (< = 0,l,a). 



(F\A) 
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(The integral li 0i „ can also be expressed as an integral over U Wo (A)\U(A) x S\(F)(C fl 
U 2 )(F)\C(A), where Si is the full stabilizer of ip v in (C fl P 2 ). In the expression above, the 
factor \ is present because the group SV^SlP^iC fl U 2 ) is of index two in this stabilizer.) 

Now, it follows from the table in the end of section 4 that 0(t) = (31 4 ). For the next step, 
we shall use this fact to prove that the integrals I^ ihUo and /° 0jJ , both vanish. 

Proposition 71. Since r is attached to the orbit (31 4 ), I™ o VQ = Iw ,v = 0- 

Proof. We have 

= j J fin(g)d^ 2 (u zu 3 (g))f T (v(y)woUog,s) dydgduo. 

U W0 (A)\U(A) (CnP 2 )(F)\C(A) 

The parabolic subgroup C fl P 2 of C = maps into the standard parabolic subgroup of 
Spu whose Levi subgroup is isomorphic to GL 5 x Sp 4 , with the unipotent radical of C fl P 2 
mapping into the unipotent radical of this parabolic subgroup in Spu. The function 9^ 
is invariant by the unipotent radical of the parabolic subgroup of 5pi 4 (A) (which lifts into 
Sp u (A)) and so is f T (w u g, s) , so this term vanishes by cuspidality of tp w . 

Next consider the integral I^, o vo - Changing variables, and using the formulas for the action 
of the Weil representation, we have 



/ 

(F\A)< 



<((0|0y|0)^<?))# <) (v(y)w u g,s) dy 



= /f (w u g, s) J 0j((O|Oy|O)iw 3 (<7)M?/i) dy 

(F\A)5 

= fr V ' K) (w u g, s) M^s(gM(o, o, 0, 0, -1, 0- 

Hence, the integral Z{J „„ is equal to 

J J <PM $^[ w </>( M o^3 (#))</>] (0,0, 0,0,-1, Ofr V '^ v) (w u g,s) dg du . 

U W0 (A)\U(A) {C<lP 2 ){F)\C(A) ^ 6i?2 

Next, let A^i denote the unipotent radical of (C fl P 2 ) and factor the integration over this 
group. Since WqNiWq 1 C £/ 4 , the function f^ v '^ [ s invariant by Ni(A). 

Now consider the partial theta function 9i(h) := J2^ F2 [u^(h)(f)}(0, 0, 0, 0, — 1, £), with 
/i G S , p 14 (A). We claim that the function n 4 1— >■ 6i(w ?) (ni)h) with ni G Ni(A),h G S , p 14 (A), 
depends only on the inner 6x6 block of 2*73(711). To see this, let Q4 denote the standard 
parabolic subgroup Spu with Levi factor isomorphic to GL 4 x Sp 6 . Note that 073(711) is upper 
triangular, and hence contained in Q 4 . Now, the function 9i is invariant on the left by the 
unipotent radical of Q4, and satisfies #i(diag(<7, Iq, t (? _1 )/i) = Jdctg0i(h) for all g G GL 4 (A) 
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and h G Sp 14 (A). This follows from the same argument used to prove lemma |4"51 The upper 
left 4x4 block of w 3 {ni) will be unipotent, and hence have determinant one. It follows that 
the function n\ 9i(zu 3 (ni)h) with n\ G Ni(A),h G Sp 1A (A), depends only on the inner 
6x6 block of 2*73(711), as claimed. 

Each element of Ni(A) can be written uniquely as 

ni(r) = X0001 (^0001)^0011 (^0011)^0111 ( r 0111)^0121 (^0121)^0122 (^0122) , 
and the projection of w 3 (ni(r)) as above onto Sp$ is 



/I 








— 2r ooi 


~2r oii 


\ 





1 











— 2rooii 








1 








— 2roooi 











1 




















1 





Vo 














1 / 



It follows that 

[w^(«otJ73(ni(r)flf))0](O, 0, 0, 0, -1, £ 6 , £7) = ^(2^ 7 roooi+2^6rooii)[wv( M ow 3 (5f))0](O, 0, 0, 0, -1, £ 6 
First, factor the integration over N 2 := ^0111^0121^0122- Thus J° UQ is equal to 

J j ^(3)^h(v 3 (s))^](0 ) I 0,O r l I 0/f*(ffloM,s) dgdu 

U W0 (A)\U(A) (CnP 2 )(F)N 2 (A)\C(A) 

where 

¥n 2 (g)= J ¥n(n 2 g)dn 2 . 

N 2 (F)\N 2 (A) 

Now expand ^ 2 (g) along £/oooi£A)oii- The term corresponding to the trivial character pro- 
duces a constant term of <p w , and therefore vanishes by cuspidality. The group S'L^ 3 per- 
mutes the nontrivial characters of this group transitively. Choose £0001 ( r oooi ^0011 ( r oon) ^ 
V'( r 'oooi) as a representative. Then the stabilizer is C/ooio- Now factoring the integration over 
^oooi^ooii(P)Woooi£^ooii(A) picks off a single term in the sum over £, and we have 



{w u g, s) 

U WQ (A)\U(A) (CnP 2 )(F)N 1 (A)\C(A) 

[u^(u zu 3 (g))(f)](0, 0, 0, 0, -1, 0, -1/2) dgdu , 
where Ni, as above, is the unipotent radical of C fl P 2 and can also be described as 
^0001^0011^2, and 

¥* 1 ^ Nl \g)= J ^■ K {ng)i) Nl (n)dn ip Nl (n) = ip{n 000 i), n G iVi(A). 

Ni(F)\Ni(A) 
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Next, the projection of ^3(^0010(^1)^0110(^2)^0120(^3)) onto Sp& is given by 



/I 



















1 


n 


2r 2 


2( ri r 2 - r 3 ) 











1 





2r 2 














1 


-ri 

















1 





\o 














1/ 



It follows that 



[^K^3^))0](o, 0,0, 0,-1, 0,-1; 



is left-invariant by t/ooio^oiio^oi2o- To show that I\ 





w ,u 



0, we will prove the following. 



Proposition 72. Since r is attached to (31 4 ), the function 

f^\g,s):= [ f T (vg,s)TP° v (v)dv 

is left-invariant by [/qioo ^0110 ^0120 (A). 



y(F)\y(A) 



Assuming proposition [72J the proof of proposition [7T] is as follows: we obtain as inner 
integration the constant term (p% where L is the unipotent radical of the maximal standard 
parabolic subgroup of Spe whose Levi part is GL 2 x SL 2 . By cuspidality this is zero. 



Proof of proposition^ Put N 3 = ^0100^0110^0120- Expand /r s) along N 3 (F)\N 3 (A). 

Under the action of SL^ 3 , there are infinitely many orbits of characters, represented by 

n 3 {r 1 ,r 2 ,r 3 ) i-> 1, n 3 {r u r 2 , r 3 ) >->• if){r 3 ), and n 3 {r u r 2 , r 3 ) ^ ip(n + ar 3 ), a e F x 

respectively, where n 3 (r h r 2 ,r 3 ) = x 01 oo(ri)xQ lw (r 2 )x 0120 (r 3 ). Now, for any a G F x , 

(73) / f^v) (n 3 (ri, r 2 , r 3 )g, s) + ar 3 ) dr 



(F\A) 3 

is a Fourier coefficient attached to the orbit (51 2 ) which is greater than (31 4 ). Hence integral 
(173"]) is zero. 

In order to prove that 



(74) 



fi V, '' Pv) (n 3 (ri, r 2 , r 3 )g, s) ip(r 3 ) dr 



(F\A)3 



vanishes, we shall relate it to Fourier coefficients attached to the unipotent orbit (3 2 1), 
which is also greater than (31 4 ). Let iY" 4 denote the unipotent subgroup of M = GSpin 7 
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which projects to 



/l 
1 



pr(n 4 ) 



n 

1 
1 



r 2 r 3 






* 

-r 5 
-r 4 



V 



* \ 

* 

-n 



1 / 



\ c SO, 



Since this projection is an isomorphism on iV 4 , we can use the matrix entries above to define 
characters of iV 4 . An integral 



J fr(n 4 g, s)ip air^j dn 4 



N 4 (F)\N 4 (A) 

is a Fourier coefficient attached to (3 2 1) if and only if the character n 4 H- ip ($^i=i a i r i) is i n 
general position, which is the case if and only if the matrix ( a\ % al ) has rank 2. Now let iV 4 
denote the subgroup of iV 4 defined by setting r 4 = r 5 = 0. Use ri, r 2 , r 3 , r 6 as coordinates on 
N® in the obvious way. We claim that 

(75) J f T (nig, s) ip (oin + a 6 r 6 ) dn 4 

iV°(F)\7V°(A) 

vanishes identically whenever aia§ ^ 0. Indeed, this integral is left-invariant by the F-points 
of two dimensional subgroup t/ooio^ono corresponding to the omitted variables r 4 and r 5 . One 
may perform a Fourier expansion along this group, and every term will produce a Fourier 
coefficient of f T attached to (3 2 1). Thus integral (1751) is zero for all choice of data, whenever 
aids 7^ 0. 

In (1751) . use the left invariance property of f T to obtain f T {n° 4 g, s) = f T (w[2]n1g, s). Conju- 
gating w[2] to the right, we obtain an integral over w[2]N^(F)\N^(A)w[2]~ 1 . This integral is 
clearly zero for all choice of data, and, for suitable nonzero a\, a§, is also an inner integration 
to integral (1711) . Thus we conclude that integral ( 1711) is zero for all choice of data. □ 



□ 



We now treat I* . Let 

^t/-o((0|0, 0, y|0)) = ij(-y 3 ), y = (y h y 2 , V3, y±, ys) e A 5 

be the unique character of U W °(A) such that 4'u w o( u )'4 ) v( v ( w o uw o 1 ))^ s trivial. 
Using the action of the Weil representation, we obtain that 

£$((010, y\, 0)uh)^ UWQ (y)dy = £ u^uh)^, 0, -1, 0, 0, f), 



(F\A)S 
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for all h G Sp u (A),u G 7£ 14 (A). Denote the left hand side by ' VuW °\uh). 

Changing variables in f T ' v then proves that 
1 



J 1 



J <pM 

U Wq (A)\U(A) SL2 2 {F)SL a 2 120 (F)(Cr\U 2 )(F)\C(A) 

^ {uoW 3 {g))fr v {w u g,s) dg du . 

Next we factor the integration over Ni := C/oooi ^0011^0111^0121^0122- This group is the 
unipotent radical of a standard parabolic subgroup of C = Sp6, having Levi subgroup 
isomorphic to GLi x Sp4. Further, it is isomorphic to the Heisenberg group Us. 

Set 



n' x {r) = Xoin(r iii)xoi2i(roi2i)a:oi22(roi22), n'[(r) = x ooi(^oooi)^oon(^oon)- 



Then one may compute w^n'^r)) and w^{n'[{r)). Each will be of the form 



h v 



with 



u G SL 2 ,h G Sp w ,b G Mat 2x io, etc., and it follows from lemma SHI that we only need the 
middle 10 x 10 block, h. The projection of w^in^ir)) onto Spio is 



h X'(r] 
h 



X'(r) 



( 2r m 


2r i2i 








-2r i22\ 





-2r m 



































-2r om 


2^0121 


V 











2r m J 



whence, by the action of the Weil representation, 

[c^(w 3 «(r)))0] (0, 0, -1, 0, 0, £1, 6) = <K-r i22 - 2£ 1 r i 2 i - 26r O iii)0(O, 0, -1, 0, 0, 6, 6) 
The projection of ^(n'^r)) onto S^io is 

/l 

1 

h"(r) := 1 



\0 



h 



Kir) 



tK(r) 



1 , 



r 0001 r 001l\ 

r 00 oi 



1 
1 





( ° 





■2r 0001^0011 





\ 










'0001 








X'{{r) := 


— 2roooi 


— 2r 0011 





'0001 


— 2roooi r oon 










— 2r on 










V 





— 2r ooi 





/ 


[c^(u7 3 (<(r)))0](O,O,- 


•1,0,0,6,6: 


= 0(0,0,- 


•1,0,0,6 


- ^0001,6 - 1 



whence 



Define 0o £ 5*(A 2 ) by 00(^1,^2) = 0(0,0, — 1, 0, 0, x±, x 2 ). Then the above shows that 
(76) [oty o ccr 3 (ni)]0(O, 0, -1, 0, 0, x 1: x 2 ) = [wjj a o t(m)]0o(^i, z 2 ). 
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for a suitable isomorphism t : N± — > Us. Here w^_ 2 denotes the Weil representation of 
7{ 5 (A) xi Sp 4 (A) defined using the additive character i/j_ 2 (x) = ip(—2x) in place of ip. By 
uniqueness of the extension of u;^_ 2 from to 7^5 (A) x Sp 4 (A), it follows that fTTol) holds 
for g G Sp 4 (A) (i.e., the preimage in C(A) of Sp\ a2 ' a3 ^ (A)) as well. 

Now, the identity component of the stabilizer of ?/^™ in Sp 4 is SL^SL® 120 . In order 
to complete our analysis of this case, we show that f T ' uW ° is left-invariant by the A points 
of this group. To prove that, and also the analogous statement related to the integrals I^ ouo , 
we prove the following. 

Proposition 77. Since r is attached to (31 4 ), the function 

fi v '^\g,s)= J f T (vg,s)Mv)dv 

V(F)\V(A) 

satisfies 

f^\hg, s) = fi V ' rv \g, s) (Wg G C(A), and h G S a (A)), 
for all a 7^ 0, where is the stabilizer of in Sp\ a2 ' a3 \ 

Proof. The group S a (A) is generated by S a (F) and the two dimensional maximal unipotent 
subgroup L (A) := S a (A) fl ^oioo^oiio^oi2o(A), so it suffices to prove that f T ' v is invariant 
by this unipotent group. Clearly, we may expand ft along I(F)\I(A). We claim that 
every term other than the constant term vanishes. 

Recall that, by hypothesis, r is attached to the unipotent orbit (31 4 ), and thus does not 
support any Fourier coefficient which is attached to the orbit (3 2 1). The set of such Fourier 
coefficients was described before equation ( 175|) . 

Let V\ denote the product of all groups U a C V except for Ui 000 . Let iV 4 denote the 
unipotent radical of the maximal parabolic subgroup of GSpin 7 with Levi isomorphic to 
GL2 x GSpin 3 , and Ri denote the product of V\ and I. Then R\ is a codimension 1 subgroup 
of iV 4 . To be precise: there is a column vector c a = (1, 0, —a) or (0, 1, 0), such that 



and 




Here we identify unipotent elements of M 4 with their images in SO7. This follows from 
the fact that I7 + re' 12 G V and I preserves ipy. Now, let ip^ be any character such that 

70 



VteJvitA) = ^lvi(A) ' and ^iIl(a) is nontrivial. We claim that 

Ri(F)\Ri(A) 

vanishes identically. Indeed, one may expand f T Rl '^ Rl Q n i?i(A)A^4(F)\A^4(A) and obtain an 
expression as a sum of terms, each of which is an integral on A^(F)\A^(A) against a character 
whose restriction to Ri is ipRi- But it is clear from the remarks above that every character 
of iV~4 which restricts to ipRi is i n general position. As explained before equation f!75|) . such 
an integral is a Fourier coefficient attached to (3 2 1), and therefore vanishes identically. □ 

It follows that with the notation as above 

U w o (A)\U(A) Ni (F)SLz 2 (F)SL° 2 120 (F)\C(A) 

^2 [^(«oG73(p))0](0, 0, -1, 0, 0, £ 6 , f 7 ) dg du . 

The inner period in this case is 



J J J <Pk{uI a(g 1 ,g 2 ) J J 9^{ua{g u g 2 )) dg x dg 2 du 




H 5 (F)XH 5 (A) SL 2 (F)\SL 2 (A) SL 2 (F)\8L 2 (A) 

where is a genuine cusp form Sp 6 (A) — > C, 6'^ 2 is a theta function on 7is(A) x Sp 4 (A) 
defined using the character tp2{%) — i)(2x) and we have identified with 

C Sp 6 . 

1 J J 

Also a : SL 2 x SL 2 — > Sp 4 is given by 

°((" M=(" 92 1 

The analysis of /* nonsquare is similar. Let 

^((0|0,0,3/|0)) = V(-2/2 + ay 4 ) 

be the unique character of U W °(A) such that Vir^o ( M )V ; y( t ' (^owWq *)) i s trivial. Then the 
integral over V(F)\V(A) picks off a partial theta series 

ef W °^ W0 \h) := 53 ^(^0(0, -1,0, a, 0,0- 

With notation as before, similar computations show that 
[w v ,(tx7 3 (n / 1 (r)))0](O, -1, 0, a, 0, f 1} f 2 ) = ^(-^oi22+2a£irom-2£ 2 r O i s i)0(O, -1, 0, a, 0, £i, 5), 
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whence 

[c^(ro 3 (n"(r)))0](O, -1, 0, a, 0, £ ls £ 2 ) = 0(0, -1, 0, a, 0, & + r 00 n, 6 - aroon). 

This is still essentially the action of the Weil representation of "H(A) x Sp A (A). Determined 
by the character ip- 2 a{x) = ip(—2ax). 

We conclude that this case does not appear to be of open orbit type. 

10. Appendix 

The purpose of this appendix is to elaborate slightly on the earlier remark that the Fourier 
coefficient (p( u ®<^u ) Q f an automorphic form cp will not, in general, be an automorphic form, 
because it will not, in general, be 3-finite. This should be something that one can prove 
using purely local methods, but here it is more convenient to point out that the existence 
of an integral representation which involves some Fourier coefficient tends to suggest that 
the Fourier coefficient is far from 3-finite. Indeed, it is, in a sense, spread out all over the 
spectrum of the operators in 3. 

To illustrate this, consider the integral representation for the exterior square L function 
of a cuspidal automorphic representation of GL 2n (A), which was given in [J-Slj . This con- 
struction consists of integrating gainst an Eisenstein series E(g,s) defined on the 
group GL n , with ip itself being a cuspform in the space of an irreducible cuspidal automor- 
phic representation of the group GLi2 n - The function ip^ J °^ u o ) inherits the rapid decay of 
the cusp form ip. Hence the integral 

<p<P°>*°o)( g )E(g,s)dg 

Z(A)GL„ (F)\GL„ (A) 

is absolutely convergent for all s where E(g,s) has no poles, uniformly for s in a compact 
set. Reversing order and making a change of variable, 

\im(p {Uo ^ u o\gexp(tX))E(g, s) dg 



I 



Z(A)GL n (F)\GL n (A) 



I 



Z(A)GL n {F)\GL n 



p- vrtuo)( g ) fim£(>exp(tX), s) dg 



for all X G gl(2, R). Consequently, even though E(g, s) is not L 2 , one may still think if this 
integral as a pairing (, ) with respect to which the elements of the center of the universal 
enveloping algebra are "self- adjoint." Now for concreteness, assume n = 2 and F = Q, and 
take A the Laplace-Beltrami operator. Then AE(g, s) = s(l — s) for all s E C Suppose that 
(p( u °>^ u a> is 3-finite. Then it is killed by a polynomial -P(A) in A. But then 

= (P(A)p, E(; S )> = ( V , P(A)E(; S)) = P{S{1 - S))((p, E(; S )) 
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for all values of s. Since P(s(l — s)) has only finitely many zeros, and (ip, E(-, s)) is nonzero 
for 9ft(s) large, we have a contradiction. 
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